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(54) System (or backing up f Oes from disk volumes on multiple nodes of a computer network 

(57) A system for bsclcing up files from disk volumes 
on multiple nodes of a computer netvrork to a common 
random-access backup storage means. As part of tfte 
backup process, duplicate files (or portions of files) may 
be identified across nodes, so tliat only a single copy of 
ttie conterrts of the duplicate files (or portions ttier eof) is 
stored in the bacl<up storage means. For eacti tjackup 
operation after the initial bad^up on a particular volume, 
only those files wtiich have changed since the previous 
badftjp are actually read from the volume and stored on 
the badoip storage means. In addition, differences 
between a file and its version in the previous bactojp 
may be computed so that only the changes to the file 
need to be written on the bacl<up storage means. All of 
these enhancements significantly reduce both the 
amount of storage and the amount of network band- 
width required for performing the backup. Even when 
the backi^a data is stored on a shared-file server, data 
privacy can be maintained by enaypting each file using 
a key generated from a fingerprint of the file contents, 
so that only users who have a copy of the file are able to 
produce the encryption key and access the file con- 
tents. To view or restore files from a backip, a user may 
mount the backup set as a disk volume with a directory 
structixe identical to that of the entire original disk vol- 
ume at the time of the backMp. 
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Description 

Field of the Invention 

The present invention relates to a system for allow- s 
ing multiple nodes on a computer network to backup 
files to a common random-access backup storage 
means. 

Background (jf the Invention u 

Backing up data and program files (often togettier 
referred to as "data" here) from computer disks has 
been a well known practice (or many years. There are 
two major reasons why data needs to be backed up. is 
The first reason is that ttne disk hardware may fail, 
resulting in an inability to access any of the valuatile 
data stored on the disk. This disastrous type of event is 
often referred to as a catastrophic failure; in this case, 
assuming that backups have been performed, the com- zc 
puter operator typically "restores" all his files from the 
most recent backup. Fortunately, new computer disks 
and controllers have become more reliable over the 
years, but the possibility of such a disaster still cannot 
t}e ignored. The second reason for backup is that users 25 
may inadvertently delete or overwrite important data 
files. This type of problem is usually much more com- 
mon than a catastrophic hardware failure, and ttie com- 
puter operator typically restores only the destroyed files 
from the backup medium (e.g., tapes) to the original so 
disk. 

In general, the backup device is a tape drive, 
although floppy disk drives and other removable disk 
drive technologies (e.g.. Bernoulli, Syquest. optical) are 
also used. Tape has the advantage of having a lower 3S 
cost per byte of storage (when considering the cost of 
the media only, ignoring the cost of the drive), and for 
that reason tape is prefened in most applicatons, par- 
ticularly those where large amounts of data are 
involved, such as network file servers. Tape is primarily « 
a sequential access medium; random accesses, while 
possible, usually require times on the order of tens of 
seconds (if not minutes), as opposed to milliseconds for 
a disk drive. Similarly, ttie time to stop and restart a 
moving tape is on the order of seconds, so it is impor- is 
tant to sipply enough data to keep the tape drive 
"streaming" in order to irsure acceptable backup per- 
formance. After a backup is completed, the tape car- 
tridge may be taken off-site for safe keeping. When the 
need arises to restore data from a given backup, the fo 
appropriate tape cartridge is re-inserted into the tape 
drive, and the user selects the file(s) to be restored, 
which are in turn retieved from the tape and written to a 
disk volume. 

The tasks of physically storing the set of tape car- ss 
fridges in a safe environment and cataloging them to 
facilitate selection of the tape(s) required for restore are 
important (and often challenging) functions of both the 
backup software and the backup administrator (i.e., the 



individual(s) responsit)le for implementing the backup 
process and policy). In addition, if the backup or restore 
operations involve multiple tapes, the ability to switch 
between tapes must be provided, either manually by a 
backup administrator or automatically using a tape juke- 
box (i.e., a robotic tape autochanger). Switching 
between tapes thus can involve a considerable direa 
cost, either for salary or tor jukebox rotsotics. as well as 
a substantial time delay, normally tens of seconds or 
more. 

In order to save backup time as well as the amount 
of tape used, various types of "incremental' backup 
strategies may be employed. For example, a common 
practice involves performing a full tiackup of all files on 
a disk volume once per week, and then backing \jq only 
the files that have changed since the last backup on 
subsequent days of the week. Another variation on this 
idea is known as "differential" backup, in which each 
partial tiacktp contains all changes since the last full 
backup instead of from the previous partial t>ackup: this 
method guarantees that only two backups (one full and 
one partial) need to be accessed to restore files as of 
the time of a particular Isackup. Since in most cases the 
amount of data that actually changes on a disk volume 
per day is a small fraction of the total, such approaches 
have the advantage of significantly reducing the backup 
"vtfindow", or amount of time required for a backup, on 
the days when an inaemental is performed. Also, it is 
often possible to fit the data from a full backup and sev- 
eral incremental backups all on a single tape cartridge, 
obviating the need for any tape switching in the days 
intervening between full backups. In the case where the 
disk volume and the tape drive are on separate comput- 
ers connected over a network, inaemental backup also 
consideratily decreases the network bandwidth require- 

While it is true that incremental backups can save 
time and media, they are also often much harder to use 
than full backups. From a user s perspective, the set of 
files included on each incremental backup is normally 
quite unrelated to how he views the contents of his disk 
volume. In other words, although certain files may have 
changed since the last backup, the disk volume still con- 
tains a complete copy of all files, changed and 
unchanged, any or which may be required to perform a 
given operation. Unfortunately, the restore software 
dealing with incremental t>ackups in the prior art typi- 
cally presents to the user a view of only the changed 
files, not a merged view of all files present on the disk at 
the time of the incremental backup. Thus, for example, it 
a user wishes to restore a given set of files, say an 
entire subdirectory, as of the date of a given incremental 
backup, he often will have to restore the files from the 
previous full backup and then each of the intervening 
incrementals in order to guarantee the correct "latest" 
copy of each file. Similarly, if the user wishes to identify 
a set of lies from the backup tapes, he normally must 
peruse several incremental/full backup sets in order to 
find all the files of interest. Once the files have been 
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selected, they may very well be spread all over the tape, 
even if they were all contiguous on the disk and the full 
backup, thus resulting in a very slow restore process. 
For these reasons, incremental backup is often used 
grudgingly at best, and not infrequently it is rejeaed in 5 
favor of always pertorming full backups. 

Another significant limitation in performing restores 
has to do with how the user may access files stored on 
the tapes. Restoring files typically involves running a 
special application, provided as part of the backup soft- /o 
ware package, that allows the user to select his tiles and 
then restore them from tape to a disk volume. Because 
the user runs the restore application infrequently, it 
presents an unfamiliar Interface to dealing with files and 
does not allow accessing the files directly with other 15 
application programs. Most users already have their 
own favorite set of applications for viewing and dealing 
with files, including word processors, file managers, 
spreadsheets, etc., so the concept of 'mounting' the 
backup image as a pseudo-disk volume to allow the 20 
user to view, select, and restore files using his own tools 
seems attractive and has been implemerrted in a few 
cases (e.g., Columbia Data's Snapback product; U.S. 
Patent Application entitled 'SYSTEM FOR BACKING 
UP COI^PUTER DISK VOLUMES,' filed October 4. 25 
1995. and assigned to the assignee of the present 
invention, incorporated herein by reference). However, 
the inherent slowness of tape in such random access 
applications makes the usefulness of this concept 
somev^at limited, and this awkwardness is particularly 30 
exacerbated if incremental backups have effectively 
spread the files out even further on the tape than they 
would be in a full tsackup. 

For a single standalone computer, the configuration 
for backup consists of adding a backup device (e.g., 35 
tape drive) to the computer. In a networked environ- 
ment, however, the situation is much more complex, and 
many configurations have been employed in an attempt 
to address the intricate tradeoffs in cost, manageability, 
and bandwidth (both tape and network). Most computer 40 
networks include nodes that are file or application send- 
ers as well as nodes that are user workstations (e.g.. 
desktop personal computers). Servers generally con- 
tain critical data for an entire conpany or department, 
so backing up the server(s) is considered an inperative 45 
task and is normally handled by a network administra- 
tor. It is not uncommon for each server to have a dedi- 
cated tape drive for backing up its disk(5), but in many 
instances a single tape drive may be used to back up 
multiple servers by sending the backup data over the so 
network. The former approach is more expensive and 
involves managing tape cartridges at more locations, 
but it avoids network bandwidth limitations in the latter 
approach that often make it impossible to keep a high- 
speed tape drive streaming with data coming over the ss 
network. Given the complexities of the various factors 
involved, including drive cost, media cost, tape drive 
speed, network bandwidth, frequency of backup, size of 
hard disks, acceptalile range of backup window, etc., it 



is not surprising that many systems utilize a mixture of 
approaches that evolves over time as technology 
progresses. 

Personal conoputer workstations on networks also 
often contain critical data The amount of such data is 
increasing as the average workstation disk capacity 
grows, due to the continuing dramatic decreases in the 
cost of disk drive capacity Nonetheless, the data from 
workstations on most netvmrks today is backed up only 
sporadically, if at all, despite the availability of several 
seemingly viable solutions For example, installing a 
standalone tape drive on each workstation could solve 
the problem in theory, but in practice there are many 
serious obstacles that limit the effectiveness of this 
approach. Among these problems are drive cost, media 
cost, end-user training cost, and the difficulty of manag- 
ing the backup tapes which are necessarily dispersed 
pfiysically throughout the organization, not to menton 
the lack of user discipline in regularly performing 
acceptable backups. 

Another- method, included as part of almost every 
netwak bad^p software package, allows the worksta- 
tion data to be backed up over the network to a shared 
backup device, thus permitting a centralized administra- 
tion of drives and media and amortizing the hardware 
cost among many users. However, despite its ready 
availability, this technique is employed in only a small 
percentage of installations, for a vanety of reasons. For 
exanple, it is not uncommon for networks to contain 
dozens or even hundreds of workstations, in which case 
there may not be enough network t>andwidth to tiackup 
all of them in a reasonable window (e.g., overnight). 
Further, the sheer volume of data involved typically 
forces the use of a tape juketxx, which greatly 
inaeases the cost of hardware and of tape manage- 
ment. Also, there can be conflicts in scheduling the use 
of the tape drive for each workstation, particularly since 
there is a need on one hand to provide data at a high 
enough rate to keep the tape drive screaming, but on 
the other hand the act of backing up typically slows 
down user response and network response. One sys- 
tem, patented by Gigatrend in U.S. Patent No. 
5.212,772. worked around part of this problem by inter- 
leaving the data from multiple workstations onto a single 
tape cartridge in an attempt to guarantee that the tape 
could continue streaming, but this approach met with lit- 
tle commercial success. Perhaps the major obstacle to 
acceptance is the simple fact that, once a user decides 
that he needs to restore data from a previous backup of 
his workstation, he does not have control of the media 
to select thetape(s) of interest, unless a very large (and 
expensive) tape jukebox is used and appropriate soft- 
ware is availat3le to manipulate the jukebox remotely. 
Manually assisting the user in this task rarely rates very 
high on the priority list of a network administrator, and it 
may also conflict with other scheduled uses of the tape 
drive(s): the result is almost invariably that the delay in 
finally accessing the data to be restored leaves both the 
end-user and the network administrator frustrated. 
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With the ever decreasing cost of disk drive capacity, 
another solution to the workstation backup problem has 
been recently employed in some networks. Ttie network 
administrator adds a large disk drive (or set of drives) to 
a file server on the network, and users simply copy files 
from their workstations to a subdirectory tree on the 
new disk. If desired, privacy of the Ijackup data can be 
insured by assigning standard network security access 
rights to each user's directory. The files placed on the 
server are backed up as part of the regular server 
backup process, providing a second level of data recov- 
ery if necessary Each user can easily access the files 
from his backup directory on the network using his own 
preferred applications, without any intervention on the 
pari of the network administrator. Note that this general 
approach could also be applied to server backup if 
desired. At current prices, it is possible to add one giga- 
byte of disk space for each user for a price comparable 
that of a low-end tape drive; while this solution may be 
more expensive on a cost per megabyte basis than oth- 
ers discussed here, the cost is nonetheless acceptable 
in certain environments. This method is not without its 
problems, such as network bandwidth constraints, need 
for user discipline in regularly lacking up all important 
files, and inability to retrieve older versions of files with- 
out accessing a tape, which typically requires adminis- 
trator assistance. However, it does overcome some key 
obstacles which are not easily addressatJe using a 
tape-only backup solution. 

It is readily observed that most workstations on a 
network contain many files with identical contents, par- 
ticularly operating system files, program files, and other 
files that are distributed as part of software packages, 
stored on the user's disk, and ne/er modified. It is also 
seems to be true that the percentage of disk contents 
occupied by such common files is increasing with time, 
particularly as disk drive capacity grows and more soft- 
ware is distributed on CD-ROMs. However, observe that 
none of the prior art backup approaches discussed 
above take advantage of these phenomena in any way. 

Summary o f the Invention 

It is the goal of the present invention to overcome 
many of the problems historically associated with back- 
ing L^) data from multiple nodes on a conputer network. 
In contrast to the prior art, the present invention pro- 
vides a lower-cost backup solution that simultaneously 
reduces network bandwidth consumption, decreases 
the time required for backup and restore, allows tor cen- 
tral administration, automates the backip process at 
user workstations, provides access to all versions of 
previous files without any administrator intervention, 
and permits the user to access files from the tjackup 
directly using his own applications. 

In the present invention, files are backed up from 
disk volumes on multiple nodes of a computer network 
to a common random-access backup storage means, 
typically a disk volume. Backups can be scheduled! 



either by the user or by the backup administrator, to 
occur automatically and indepenoently (or each node. 
As part of the backup process, duplicate files (or por- 
tions of files) may be identified across nodes, so that 
s only a single copy of the contents of the duplicate files 
(or portions thereof) is stored in the backup storage 
means. The preferred embodiment includes a search 
method for identifying duplicate files that is extremely 
efficient in its use of network bandwidth even when mil- 
Tc lions of files have been added to the backup system. For 
each backup operation after the initial backup on a par- 
ticular volume, only those files which have changed 
since the previous backup need to be read from the vol- 
ume and stored on the backup storage means; poimers 
;5 to the contents of unmodified files are stored along with 
the directory information for the backup. In addition, dif- 
ferences between a fie and its version in the previous 
backup may be computed so that only the changes to 
the file need to be written on the backup storage means. 
20 and almost all data written to the backup storage means 
is compressed using a lossless compression algoritttm. 
Each of these "data redLction" enhancements signifi- 
cantly decreases both the amount of storage and the 
amount of network tiandwidth required for performing 
25 the backup. In fact, the data reduction is effective 
enough in most instances to lower the amount of stor- 
age required to the point where the system cost of using 
disk drives as the backup staage means is less expen- 
sive than the cost of conventional tape backup systems 
30 in such environments, particularly given the rapidly 
declining cost of disk storage. 

Even when the backup data is stored on an openly 
accessible shared-file server, data privacy can be main- 
tained by encrypting the contents d each file using an 
35 encryption key generated from a hash function of the file 
contents, so that only users who once backed up a copy 
of the file are able to produce the encryption key and 
access the file contents. 

To view or restore riles from a tMCkup. a user may 
40 mount the tDackup set as a real-time (i.e., with disk 
access times, not tape access times) temporary disk 
volume with a directory structure identical to that of the 
entire original disk volume at the time of the backup. 
The user may then access the files directiy using his 
45 own applications, without first having to copy them using 
a separate restore program. The backup disk volume 
may be mounted in a read-only nxxle; alternatively, 
write access can be provided to allow transient modtfi- 
cations to the files, although all such modificationE are 
50 normally tost once the backup volume is unmourted. 

Brief Description of tha Hrawin^e 

A preferred embodiment of the present invention is 
55 illustrated in and by the following drawings, in which like 
reference numerals indicate like parts and in which: 

FIGURE 1, is a block dagram illusti-ating a typical 
network configuration for the backup system of ttie 



4 



EP 0 774 715 A1 



present invention; 

FIGURE 2 is a diagram illustrating a typical direc- 
tory structure where backup files are stored, 
FIGURE 3 is a block diagram illustrating the types 
of files contained in the backup directories of the f 
present invention ;- 

FIGURE 4 is a Backus-Naur Form (BNF) descrip- 
tion of the format of the directory entries in a 
backup directory file in accordance with the present 
invention; ,a 
FIGURE 5 is an assembly language description of a 
specific example of the direaory entry format 
defined in FIGURE 4; 

FIGURE 6 is a block diagram of the layout of a 
backup data file in accordance with the present is 
invention; 

FIGURE 7 is a BNF description of the format of a 
backup data file in accordance with the present 



FIGURE 8 is a block diagram illustrating an exam- 
ple o< a (seBkPts) record in a backup data file, in 
accordance with the present invention; 
FIGURE 9 is a diagram illustrating the layout of a 
global directory database file in accordance with 
the present invention; 

FIGURE 1 0 Is a diagram illustrating the data struc- 
tures used in searching the first level of the global 
directory database in accordance with the present 
invention; and 

FIGURE 1 1 is a diagram illustrating how user pass- ; 
words are used to access encryption keys to 
access backup data in accordance with the present 
invention. 

Detailed Pescrintion of thP PrpfPrred Frnhnriirporyt 3 

The preferred ent)odiment of the present invention 
uses disk space on a network file server (or servers) as 
Its backup storage means. Each client workstation is 
responsible for copying the backup data to a preas- 
signed location or directory on the file server, as well as 
for searching the backup "database" to identify dupli- 
cate files across users and to compute the differences 
(or deltas) between file versions. Thus, the preferred 
embodiment is not truly a clienVserver system, although 
certain housekeeping functions essential to perform- 
ance and security need to be performed by an Agent 
task, which may run on any network node, including the 
file server itself. 

In an alternate embodiment, the backup storage 
means consists of disk space on an application server 
(the backup server). The network nodes communicate 
with the backup server in a traditional client/server par- 
adfgm. The Agent functions are performed by the 
backup server. This embodiment provides slightly 
higher security than the preferred embodiment, but it 
normally costs more because of the need for a separate 
server, although ft may be possible to amortize this cost 
somewhat if the backup server provkies other applica- 



tion services. Such an embodiment also tends to con- 
centrate the computing load (e.g., identifying duplicate 
files) on the server, which may affect the scalability of 
this approach, although there are simple ways to distrib- 
ute more of the computational load across the client 
nodes if desired, which are readily apparent to those of 
ordinary skill in the art. There are also many other pos- 
sit>le embodiments, consisting of various flavors of 
hybrids of the file server and application server 
approaches, which would fall within the scope of the 
present invention. 

In yet another embodiment, the backup storage 
means incorporates hierarchical storage management 
(HSM), in which files that have not been accessed for a 
long time are migrated from disk to a secondary storage 
means, such as tape or optical disk. The main purpose 
of HQM is to save on storage costs for very large stor- 
age systems by provkiing the management tools that 
allow the migration to be transparent to the system, 
except for the additional delay in accessing some files. 
Use of any form of HSM in conjunction with the backup 
storage means of the present invention does not signifi- 
cantly affect any of the concepts discussed here. How- 
ever, care must be taken not to impair performance of 
the backup and restore opera«c»is, since delays 
incuned in accessing secondary storage may render 
the system much less usable. Indeed, it would be fairly 
simple to identify portions of the contents of backup 
data and directory files of the present invention which 
could be migrated to secondary storage without 
adversely affecting backup performance. Fortunately, in 
most cases, the data reduction methods of the present 
invention are sufficiently powerful to keep disk storage 
HSM ^° ^" ^'^'^^P*^® '^'^l without using 

1. Backup Process 



In the prefened embodiment, as shown in FIGURE 
« 1 . the nodes to be backed up may be either worksta- 
tions 102, desktop peisonal computers 103, laptop 
computers 104, or other servers 105 on the network. All 
communication is accomplished by creating or modify- 
ing files over the network 106 on the backup storage 
45 101. As shown in FIGURE 2. each node is assigned two 
dirertories, a user directory and a system directory, on 
the backup storage means 101, which is contained in 
the disk volumes of the network file server 100. The 
node has network write access to its user directory 
so (e.g., \BACKUP\USERS\USER2. 125 in FIGURE 2). 
where it posts backup data. A backup administrator con- 
figures the backup system using administrator software 
functions prwided as part of the product. A backup 
Agent process 108, which runs on a network node 107 
55 selected by the backup administrator, migrates the 
posted files to the system directory (eg 
VBACKUP\SYSTEM\USER2, 128 in FIGURE 2). This 
system directory has network rights assigned to make it 
read-only to all nodes (except the Agent 1 08). so that no 
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user node can corrupt the migrated backup data, inten- 
tionally or inadvertently. While it would not be strictly 
necessary to use two directories, data integrity is signif- 
icantly improved in the shared-file environment of the 
preferred embodiment using this approach. 5 

In the preferred embodiment, the Agent 108 has 
read-write access to all the directories shown in FIG- 
URE 2. Each user is given read-only access to alt the 
directories under \BACKUP\SYSTEM 122, txit he has 
no access to any of the directories under rc 
\BACKUP\USERS 121. other than his own directory 
(e.g. \BACKUP\USERS\USER2, 125 in FIGURE 2). to 
which he has read-write access. Limiting access to the 
posting directories in this fashion furttier increases 
security, since no user can inadvertently corrupt another 15 
user's posted backup files before they are migrated. 
However, since in the preferred embodiment all backup 
files are encrypted and have checksums that can be 
used to detect corruption.it would also be possible (and 
probably easier, from the viewpoint of the network so 
administrator] in an alternate errtxxliment to give all 
users read-write access to all drectories under 
\BACKUP\USERS 121 wittiout significantly compromis- 
ing security, assuming reasonably well-behaved users. 
The Agent 108 checks the integrity of each backup fie 25 
while migrating it; if any errors are detected, the file(s) 
are not migrated, thus maintaining the integrity of the 
data on the \BACKUP\SYSTEI^ directories. Observe 
that this general approach of the preferred embodiment, 
using network access rights and an Agent 108, results so 
in much higher levels of data security and integrity than 
in a conventional shared-file application, where each cli- 
ent node typically has full read-write access to the 
shared files, which are therefore much more susceptitie 
to corruption. 35 

FIGURE 3 shows the main types of files (as well as 
some of their inter-relationships) created as part of the 
backup system of the preferred embodiment. During the 
backup of a disk volume on a rxxle, the backup process 
of the preferred embodiment separates all files on the 40 
source disk volume into four categories: new. 
unchanged, updated, and modified. New files are those 
which did not exist on the same directory at the time of 
the previous backip. Unchanged files existed at the 
time of the last Isackup and have not changed since that 4S 
time (e.g., they still have the same time. date, and size). 
Updated files are files that had been unchanged for 
more than days as of the ime of the previous 
backup, where is a user-selectable option (typically 
in the range of U-90 days), but which have been so 
changed since the last bacfep. All other files are classi- 
fied as modified. When the first backup of a given vol- 
ume is performed, all files are classified as new. For 
each new or updated file, the backup software searches 
through a global directory datatase 1 45 for a matching ss 
file. The global directory database 145 is created and 
maintained by the Agent process 108 in the directory 
\BACKUP\SYSTEM\GLOBAL 127. Each tmelhe Agent 
108 migrates a backup set from the \BACKUP\USERS 
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path 121 to \BACKUP\SYSTEM path 122, it searches 
for new and updated files in the backup set and adds 
them to the global directory database 145. If a matching 
file is found in the database, a reference to the contents 
of that file is stored instead of the file data itself, as 
described below. Similarly, for unchanged files, only a 
reference to the previous file contents is stored. 

In order to minimize search time and bandwidth, it 
is believed preferable not to conduct a search through 
the global directory database 145 for moditied files. For 
the same reasons, and to minimize the growth of the 
datatase, modified tiles are not added to the global 
directory dataisase 1 45. Instead, the contents of modi- 
fied files are stored in the backup by computing the dif- 
ferences from the most recent version(s) of the file and 
saving either the differences or the new version in its 
entirety, whichever is smaller. Differences may be com- 
puted and represented in any manner known to those of 
ordinary skill in the art. The updated category, whwh 
can be thought of as a special user-defined subset of 
the modified category, serves to identify duplicate files 
across users which are updated on an infrequent basis. 
One common instance of such files woukJ be the new 
version of the exocutables of a word processor or some 
other popular application. Note that setting Ny to zero 
eliminates the modified category (i.e.. all changed files 
are in the ipdated category], while setting to infinity 
eliminates the updated category (i.e.. all changed files 
are in the modified category). 

1.1. Backup Directory Files 

TTie backup process of the preferred entKximent 
actually creates two files containing information about 
each backup set: a backup directory file (e.g., 143). and 
a backup data file (e.g., 144). In an alternate embodi- 
ment, these files could be combined into a single file. 
The contents of the backup directory file indicate the 
directory structure of the source disk volume, as well as 
pointers into the backup data tiles (e.g., 144, 149, and 
backup data files of other users) indicating where the 
data for each file is to be found. One key feature of the 
present invention is the data reduction achieved by 
duplicating pointers to data and directory informaton 
instead of duplicating the information itself, including 
referencing duplicate information across users. To 
explain the role of the backup directory file(s) in accom- 
plishing mis data reduction in the preferred embodi- 
ment, a description of key portions of the badaip 
directory file (e.g.. 143) for a DOS disk volume is given 
in FIGURE 4 in Backus-Naur Form (BNF), which is a 
well known formal language technique (lor example, 
see NicWaus Wirth, Aloorithms -h D ata Strueturgs = Prrv 
orams. 1976. pp. 281-291). Before discussing the con- 
tents of FIGURE 4, we will explicitly define the 
conventions of our BNR since there are slight variations 
in syntax from one author to the next. Non-terminals are 
enclosed in angle brackets (e.g.. ,). The ;== 

symbol indicates a fomial definition. Terminals are irdi- 
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cated as single binary digits {q or or as hexadecimal 
quantities using C-like syntax; o^^u 8-bit bytes, 
oxuuuu for 16-bit words, and oxuuuuuuuu for 32-bit 
dwords Ranges of terminal values are indicated as two 
terminal quantities with two periods in between; e.g., 
0x00 .. cxFE- The : character is a meta-symbol indicating 
"one or the other", while brackets 0 indicate an optional 
field, and an asterisk (•) indicates one or more repeti- 
tions of the field. Thus, for example, U„„^D,m„ ,]■ indi- 
cates zero or more of the non-terminal (exiernoirhem - ■ 
The double slash // indicates a comment to the end of 
the line. 

FIGURE 4 defines the format of the directory infor- 
mation in a backup directory file (e.g., 143). At 200. the 
<voiumeDirinto) Section of the file is defined to be a series 
0' (subdirHieLisi) records 201, followed by a separate list 

0'(9x.emDirl,em) records 220. Each (saWrFilaUs.) record 

201 contains the directory entries for the files and sub- 
directories in a single directory. In particular, as shown 
at 201. each (sutxirrFii«Listi record consists of a series of 
(liioErtry) a"d (sybdirEmry) records 207, 208 (containing 
the directory entries for files and subdirectories, respec- 
tively, found in the associated directory) and is termi- 
■ "3*^ by an (endoiList) marker 202 (for example, a zero 
byte). The (endOfListi marker 202 is followed by 
Count) 203. which is a variable-length encoded in1ege"r 
siemcoum ■ 204, representing the number of .B^orrOiriw > 
records 220 associated with this directory. The partc^- 
lar encoding (204, 205. 206) of .^^coun.) used in the 
preferred embodiment is not important; many simple i 
alternate encodings would serve equally well, though it 
IS usually desirable for the encoding to take advantage 
of the fact that small counts are much more common 
than large ones in order to minimize the average code 
size. In an alternate embodiment, the (amrnnim i a 
records 220 associated with each <subdirFiioUst) 201 
could be stored immediately after the ,ext9mccun,. field 
<^03 instead of in a separate section, as shown at 200- 
the preferred embodiment keeps these sections sepa- 
rate as a slight optimization, allowing the Agent 108 to * 
scan through the entire (,«„nD,ri,«m) list 220 qui*ly to 
see which externa) hems are referenced without the 
overhead of parsing the (subdirFiisListi section 201 . 

In the preferred embodiment, the directory tree is 
represented implicitly by placing the <3uMirFi,oUs.. « 
records 201 in a conventional d^th-first ordering In 
other words, each time a subdirectory is encountered 
while -walking" the directory tree on disk, the ,,^;.„„,, 
record 208 for that subdirectory is appended tolie cur 
rent (siirtj,p.|^Lisi) 201. and a token representing that so 
subdirectory is pushed onto a temporary internal stack. 
When processing of the current subdirectory is com- 
pleted, the te„doiU3t. and te„errCoun,. fields 202, 203 are 
appended as shown at 201 , and processing then contin- 
ues in the subdirectory represented by the token which ss 
IS popped off the internal stack. If the stack is empty, the 
entire directory tree is complete. In an afternate embod- 
iment, another tree ordering (e.g., breadth-first) could 
be used to achieve similar resuls. 



Each explicit aiieEoTy record 207 is assigned a 
directory item number 223) that is unique 

across all backup sets for that user, in the preferred 
embodiment, this number is an incrementing Si-bit 
5 quantity as shown at 223 in FIGURE 4; this size is suffi- 
cient since it allows, for example, for up to io backups 
per day. each with 10,000 changed files, for a period of 
over 50 years before overflow could occur. Of course a 
larger quantity of bits could be used if necessary The 
w range of (di,itamNLim > values 223 used is defined else- 
where in the backup directory file (e.g., 143) ■ in order to 
save space, each <,|,e„„^„ 207 in the ; volume D„inio. 
record 200 is implicitly assigned the next ui,i,8„Nt,^ ,223 
in the range, so that no explicit wintomNum > 223 n^s to 
15 stored along with each „,ieE„,ry) 207. During the backup 
process, when an unchanged file is found, instead of 
duplicating the „,teEmry. 207 tor that file, a reference 
may be added to the previous , 207 by including 

rtS (dirllemNum) 223 in the („,ernD,rltem> 220 list aSSOQ- 

M ated with the current directory in the fairly common 
case where muftiple unchanged files are found with 
consecutive <diritomNum > values 223, in order to save 
space this sequence is indicated by a (manyii.msv record 
222. consisting, for example, of a one-bittag (7)!a 31-bit 
'5 (diriienNum) 223, and an (hsmcounii record 204 whicti 
represents the number of consecutive external „i,,p„, , 
records 207 referenced. Othenvise the r r. 
220 is represents as a 221, conlS'T^ 

one-bit tag (o) to distinguish this field 222 from a 
0 field, followed by the 31-brt 223 oT'the 

referenced <„, 207. Note that the , field 

203 in the preferred embodiment counts the number of 
(axisrnDirtem ) 220 fecords, not the total number of „:„c^ 
fry , records 207 referenced thereby. 

JiieEntry ) arxJ (subdirEntry ) records are of variable 
length and consist of several fields, as shown at 207 
and 208 in FIGURE 4. These fields are dictated by the 
attributes of the underlying file system; tor purposes of 
illustration, the definitions of FIGURE 4 include 
attributes required for a DOS FAT file system, but obvi- 
ous, modifications can be made to the ir,^^^^y, defini- 
tion 207 to allow for different attributes in different file 
systems (e.g.. Macintosh, OS/2 HPFS. NetWare, etc.) 
The header of the backup directory file (e.g., 143) in the 
preferred embodiment contains a field specifying the 
source file system and thus indicates the particular tor- 
mat of the (fiieentry, records 207 in this backup In the 
example of FIGURE 4, the first field is the <,i„K,,„», 
record, defined at 212. which is a zero-terminated Tri- 
able length character string as defined at 219) 
representing the name of the file. Next comes the (,(1,^,. 
irib > field 209, virhich is a single byte containing attribute 
bits, such as read-only, directory, hidden, system etc 
The file modification time , 210 follows: this 32- 
bit quantity includes both the time and date when the file 
was last modified. In mae advanced file systems sev- 
eral other time values could be added here, such as last 
access time, creation time, etc. The ,i„si„, field 21 1 is 
a 32-bit quantity representing the size or the file in 
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bytes. Finally, the 

(fiiaiD) 214 field indicates where the 
file data associated with this file can be found. As shown 
at 214, this information includes a luserinoex.i 216 and a 
(liieindat) 215. Each user on the backup system has a 
unique user number {usermdsxi 2i6, which is a 16-bit 5 
quantity in the preferred embodiment. Similarly, each 
file is assigned a unique number, similar to the (dritsm- 
Mum, 223 for directory items: this ((iieindex) 215 is a 32- 
bit quantity in the preferred emtjodiment. The two fields 
that make up the (meiD) 214 can be used to locate the rc 
appropriate backup data file (e.g., 148) containing the 
file data, as will be discussed below. The (subdirEniry> 
record 208 consists of the same first three fields as the 
ifiieErtry) record 207. except that (fiisTime ) 'ieW 210 indi- 
cates the directay creation time. In an alternate embed- 75 
imem. each <(iieEntry: record 207 also contains a 
oastversion) 'ield, which is a (cjiritemNum :• 223 that directly 
references the (meEntry) 207 for ttie previous version of 
the file: this technique provides a linked list of all unique 
versions of the file, which could be reconstructed con- 20 
sideraUy.more slowty by reading and parsing all backup 
directory files (including those where the file was 
unchanged). 

In the preferred en*odiment, there is no way to ref- 
erence unchanged (subdirEntry> records 208 from previ- 25 
ous backups. In other words, the entire tree of 
directories must be explicitly represented in each 
backup directory file (e.g.. 143). although the files within 
those directories can be incorporated by references in 
the iaxiornDiriiam) section 220, as discussBd above This 30 
somewhat artiitrary decision in the prefened embodi- 
ment was made to simplify the backup and restore logic 
slightly at a small cost in the size of some backup direc- 
tory files, but in an alternate emtxxliment it would be 
simple to allow referencing unchanged subdirectories 35 
(and entire file/subdirectory trees). The size of the 
backup directory file (e.g.. 143) is normally, a small frac- 
tion of the size of the backup data file (e.g., 144), and 
the contrltxrtion from the subdirectory entries alone to 
the size of the backup directory file is normally not sig- 40 
nificant. so this issue appears to be of very minor con- 
cern at most. 

A somewhat related and possibly.greater concern 
is, the fact that, according to the defiritions of FIGURE 
4, there is no limit on the number of external backup « 
directory files (e.g., 14S) referenced by a given backup 
directory file (e.g.. 143). If this number were to grow 
without bound, when it came time to perform a restore, 
the amount of time required to reconstruct the directory 
tree could be quite large, even though all the backup so 
directory files are on disk. In practice, in the prefened 
embodiment, there is a limit imposed at backup tme 
during the construction of the backup directory file (e.g.. 
143). on the number Nq of external backnj directory 
files (e.g.. 148) which can be referenced. Typically ttis 55 
number is set in the range of No = 5-20 files. The result 
is that the (fiisEntry) record 207 for an unchanged file is 
explicitly re-induded about every Nq backups. This pro- 
duces only a tiny increase in the overall storage require- 



ments on the backup storage 101. but it guarantees a 
reasonable response time during the restore operation. 

Note that, in the preferred embodiment, each user s 
backup files may only reference lexiemDiriam records 
220 from his own previous backups, not from other 
backups of other users. This decision, which results in a 
very minor cost in overall storage requirements, stems 
from the desire to maintain privacy of all user directory 
information. As we will see below, the contents of each 
backup directory file are encrypted so that no other user 
can even see the names, sizes, dates, or attritiutes of 
another's files, which might in themselves compromise 
privacy even without access to the actual file contents. 
By contrast, the data contents of files that are backed up 
can be stiared between users, with privacy insured via a 
unique encryption key protocol discussed below. If the 
size of the backup directory files became a significant 
issue in an alternate embodiment (e.g., a new type of 
file system), techniques similar to those used for data 
COM be applied to directory entries to save space if 
desired. For the file systems of interest to the preferred 
embodiment at this time, however, there appears to be 
no compelling need to minimize the size of the backup 
directory files any further 

To illustrate further the meaning of the BNF defini- 
tions in FIGURE 4. FIGURE 5 contains a small example 
of the format of a ,voiumeDirinta> record 200. The format 
is that of 8086 assembly language, which allows for very 
flexible (if somewhat primitive) output of variable length 
fields. A semicolon (:) serves as a comment to end of 
line. TTie following directives are used to control output: 

jt, = emit 8-bit byte(s) 

= emit 1 6-bit wo»d(s) 

aa = emit 32-bit dworcl(s) 

Hex constants end in H (e.g., soooooooh)- Sweral fields 
in the example are left undefined using the (?) expres- 
sion: tor example, the tile time/dates are unspecified 
because the particular 6mes are not of interest for pur- 
poses of this illustration. In general, in order to clarify 
the usage, each line Is followed by a comment contain- 
ing ttie BNF non-termnal(s) corresponding to that line. 
The line-byline comments refer directly to the BNF of 
FIGURE 4, which has boon described in detail above. 
Note that the <exiernDiritem ) 'ist, Starting at 356, does not 
give any indication of the actual contents of the (fteEmryi 
records referenced: it is necessary to read and parse 
the contents of the separate referenced backup direc- 
tory file(s) in order to obtain the directory entries. 

In the preferred embodiment, each t«ckup direc- 
tory file (e.g.. 143) contains several other sectiofB. 
These sections are described briefly here. They gener- 
ally involve well-understood techniques that are used in 
many other backup products, and are therefore readily 
understood by those of ordinary skill in the art. However, 
it is useful to give a brief explanation of the contents and 
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purpose of these other sections to give a broader back- 
ground context tor the present inventions. Each section 
is covered by a checksum or CRC to allow for corruption 
checks, and many of the sections (including the (yoiu- 
neDirinfo) section 200) are compressed using, well 
known compression "techniques, such as those 
described in U.S. Patent 5.016,009, or U.S. Patent 
Application Serial No. 97/927,343 (filed August 19, 
1992, entitled -DATA COMPRESSION APPARATUS 
AND METHOD USING MATCHING STRING SEARCH- 
ING AND HUFFMAN ENCODING'), both of which are 
assigned to the assignee of the present invention and 
both of wrhich are incorporated herein by reference. In 
addition, each section (other than the header) is 
encrypted using a private key encryption scheme, such 
as the Data Encryption Standard (DES) or RSA's well 
known RC2 or RC4 algorithms: the key management 
protocol for this encryption is discussed in detail below. 
Finally some primitive error correction ability is incorpo- 
rated into each file by appending a section of overall 
parity sectors at the end of the file contents. 

Each backup directory file in the preferred embodi- 
ment begins with a header which includes a signature 
' and creation time stanp, as well information on file for- 
mat version, file size, and pointers that identify the loca- 
tion and size of all other sections. A TskipDescription" 
section contains descriptive information about the 
baclojp operation, including a user-generated annota- 
tion string, time of the backup, count of new files and 
bytes, ranges of new (dirnamNum) an^^ (fiiaindei) records 
223, 215 generated, a backup file number, the luserindoxi 
216, and a specification of the source volume for the 
backup. A "dirlndexRange" section is a small variatale- 
length record which identifies the exact set of new i^^,. 
iiemNura) values 223 included in the file, from which the 
(diriieniNum) 223 assignments are made in (v-oiumaOirinto) 
209, as discussed previously; normally finere is only a 
single contiguous range of values, but it is possible after 
the Agent 108 has performed a consolidation operation 
(discussed below) for multiple non-contiguous ranges to 
exist in a single file. A "dirttemPr section contains an 
array of pointers into (voiumaDirinio ) 200. one pointer for 
sach (fiiaEniryi 207. This section is actually redundant 
and can be reconstructed by parsing (voiumeOirinto) 200; 
together with the "dirlndexRange" section, it serves to 
speed access to a (fiieEntry) record 207 from a separate 
backup directory file via a (dirhemNumi reference 223. 
Finally a "fileDecryptKey" section contains a private 
encryption key (e.g., for DES) that is used for decryption 
of the data file contents. There is one key in this section 
for each (nieEnuy) 207 in (voiu,„eDi,info > 200; in fact, con- 
ceptually this key is part of the jfaeEntryi record 207, but 
it is placed in a separate section in the prefen'ed embod- 
iment solely because including it directly in the <fiioEniry> 
207 would lower the compression ratio of the (voiumaOir- 
Wo > section 200. 

There are many equivalent ways to organize the 
information in the backup directory file to achieve similar 
results. The particular record formats of preferred 



embodiment described here are not intended to limit the 
scope of the present invention. 

1.2. Backup Data Files 

s 

The backup data file (e.g., 144) contains data from 
the files included m the backup set. Some of this data 
may be represented by references into Other backup 
data files from previous backups (e g., 149), either from 
10 this user or another user. Each unique file included in 
the backup data file is assigned a ;fiieindfix.> 215, which is 
a 32 -bit number in the prefen^ed embodiment and which 
Is used to reference that file. Observe that there is no 
one-to-one correspondence between (oiritemNum - 223 
and (fiiaind«) 215 values. For example, if user A has an 
exact copy of a file that has already been backed up by 
user B, user A's ttjieEntry.> 207 will contain the identical 
flilelD) 214 (i.e., (fiieindex) 215 and .luserindax! 216) as 

user B's, but they will fiave distinct (diritemNum > values 

so 223, which are not shared between users in the pre- 
fened embodiment, as discussed previously f^ost of 
the data in a backup data file is compressed, the data 
from each file included in the backup set is encrypted 
using a iile-specrfic key (stored in the lileDecryptKey" 

25 section of a backup directory file, such as 143) instead 
of a private user key; in other words, multiple encrypton 
keys are generally used in each backup data file. The 
key management protocol used to guarantee data pri- 
vacy will be explained in detail below, but the net result 

30 is that, in the preferred embodiment, the contents of 
each backup data file are effectively publicly available, 
by contrast to the contents of the backup directory file, 
which are encrypted with a private, user-specific key 
A high-level block diagram of the layout of a tjackup 

35 data file {e.g., 144) of the preferred embodiment is 
shown in FIGURE 6. The file consists Of four main sec- 
tions, of which the Data Blocks section 161 is typically 
by far the largest since it contains the actual contents of 
the files included in the backup set. Because of its size, 

<o the Data Blocks seaion 1 61 comes directly after the 
fixed-size Header 160 in the preferred embodiment so 
that file data can be written directly into the backup data 
file without ever having to move the data blocks again. 
The Header section 160 contains a signature and crea- 

4s tion time stamp, as well as information on file format ver- 
sion, file size, and a pointer 162 to the FilelnfoPtrs 
section 178. Uke the backup directory file, the backup 
data file may also contain parity sectors to allow simple 
en-or con-oction in the case where small disk flaws 

50 develop on seaors on the backup storage means 101. 
The FilelnfoPtrs section 178 contains a variatsle size 
record indicating the exact set of (fiiemaax) values 215 
represented in the backup data file; this record is very 
analogous to the "dirlndexRange" section of the fcackup 

55 directory file discussed above, and typically consists of 
only a single contiguous range of values. The rest of the 
FilelnfoPtrs section 1 78 contains an array of fixed size 
entries (e.g., 181). one entry per (rMrr)ax> value 215 
represented in the range. Each entry contains a pointer 
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(e.g., 179) into the Filelnfo section 175, where there is which may reside either In this backup data file {e.g.. 
one variaUe-sized entry (e.g., 1 76) per file. In addition, 1 63) or an "external" backup data f ile ( e. g. , 1 49). These 
each entry (e.g., 181) in the FilelnfoPtrs section con- blocks are then read, decrypted, and decompressed to 
tains othertile-specific information (such as file size and pro/ide the original file contents. It can be seen fairly 
a CRC over the initial blocks of the file contents) s easily that accessing any porton of the file contents 
required to enter each new or updated file into the glo- requires only a handful of disk accesses. AKhough the 
bal direaory database 145. Each Filelnfo entry (e.g., number of accesses is probably larger than would be 
176) contains information on the contents of the file. necessary to access a file on a "normar file system, it is 
including a variable length array of pointers (e.g., 1 73) still small enough that the access time during restore is 
into the DataBlock section 161 or into the contents ol ,c measured in milliseconds (or tenths of seconds at 
files contained in other backup data files. worst), not the tens of seconds or the minutes normally 
For example, FIGURE 6 illustrates some details of associated with restore operations from conventional 
two files contained in the backup set. The FilelntoPtr tape backups. Clearly, in order to optimize restore 
entry 1 81 fa File A includes a pointer 1 79 to the Filelnfo "access" time, the restore software may include an inlel- 
entry 176 for File A. This entry 176 contains a set of is ligent caching algorithm for the contents of the backup 
pointers 173, including pointers 164 and 165 to data directory and backup data files, 
blocks 1 63 and 165, respectively, in the DataBlocks sec- Given this high-level understanding of the various 
tion 161. as well as pointer(s) including 167 to data sections of a backup data file, consider FIGURE 7, 
blocks in other backup data files. All data blocks (includ- which contains a set of BNF definitions giving consider- 
ing 163 and 165) in this backup data file associated with sc aWy more detail on the format of some of these sections 
File A aro encrypted using the encryption key for file A, At 400. the entire file <bKupOaiaFiie ) defined to consist 
as shown at 168; this key is stored in the lileDe- of the four main sections discussed above (ihoador) 
ayptKey- section of the backup directory fil6(s) contain- section 401. <(jaaBiock) section 405, (tuemio) section 
'"3 a (tieEniryi 207 whose (fiieiQ] 214 references File A. 408. and (fintoPtra) section 432); the remainder of FIG- 
Similarly. the FilelnfoPtr entry 182 for File B includes a ss URE 7 describes the contents of these sections, 
pointer 180 to the Filelnfo entry 177 for File B. This entry The relevant portions of the <h««d.r) section are 
177 contains a set of pointers 174, including pointer 170 listed at 401. In particular, the (finioPtottB*!) <'eW is 
to data block 171 in the DataBlock section 161 , as well defined at 402 as a pointer (32-bits in the preferred 
as poinler(s) including 172 to data blocks in other embodiment) to the start of the (nntePtnn section 432. 
backup data files. All data blocks (including 1 71) in this 3C The <indaxRargeCnt) 'ield is defined at 403 as a count of 
backup data file associated with File B are encrypted the number of (indaxRango) entries 433 in the (imtoPirs) 
using the encryption key for file B, as shown at 169. section 432. 

Given a anaio , 21 4 and the decryption key, it is rel- At 405 and 405. each (daiisiock > 'S defhed to be a 
ativety straightfonward to extract the file contents to varialjle-length array of 8-bit bytes. In the preferred 
"restore" a file. First, a search is performed through the 35 embodimertt, each idaiaBiodt) 'WS starts at an offset in 
backup data files of the user identified by <uMrindei; 216 the backup data file (e.g.. 144) which is a multiple of 4. 
for the backup data file containing the {u^Mtx' 215 of so that the offset can be encoded in 30-bits. TTiis con- 
interest. This search can be easily performed, because vention allows slightly tighter packing of the ;s„KPoint) 
the Header 160 and FilelnfoPtrs 178, which contains fields 416 at a very minor cost in the overall size of the 
the range of (fiiain<i««) values 2l 5, are not encrypted. In 4C backup data file, but this optimization is in no way critical 
the prefen-ed embodiment, the search can normally be to the invention. Each dataWock may be compressed 
performed even more quicWy because the Agent 108. (indicated by <packFiag> 422 of the associated <<jaiaB- 
as part of the migratkjn process of backup data files K^^Ptr) 419) and then encrypted. The enwyption key for 
from\BACKUP\USERS 121 to\BACKUP\SYSTEM 122, each (d^toBiodo "WS is not kept in the backup data file 
builds a special Index Range Lookup file (e.g.. 151 of 4S (e.g., 144); as discussed previously, the keys reside in 
FIGURE 3). This file, which is redundant in the sense encrypted from in the backup directory f ile{s) which ref- 
than it can always be re-built from the contents of the erence the associated file data blocks. Typically the u,,. 
backup data and directory files, includes a table which gBio*) section 406 is by far the largest section in the 
maps index ranges into backup data file names and backup data file. As part of the enayption process, a 
which is an-anged for a fast binary search. With the sc checksum is appended to each (jjataBio*) ^5 ofder 
appropriate backup data file identified, this file is to facilitate a quick check for corrupbon. either ol the 
opened, and the pointer 1 62 to the FilelnfoPtrs sections tjlock itself or of the pointer to if. 
is read from the Header 160. The index range record of Definition of the ((inteps,) section begins at 432. In 
the FilelnfoPtrs section 161 is then scanned to identify particular, this section consists of two variable-length 
which pointer corresponds to the given <,i|.|^„, 215; ss arrays of <i«texR.ng.) 433 and „ii„nteO«u» records 436. 
that pointer (e.g., 179) IS then used to index the Filelnfo As discussed previously, the nunber of r,niiaxntnaa> 
entry (e.g., 1 76) for the file of interest. From the Filelnfo records 433 is indicated by the (i„d«F».n»cni > field 403 
entry, pointers (e.g.. 173) are found to the data blocks of the headaD^OI. Given the (ind«xR«ngecnt) value 403 
corresponding to each portion of the fie of interest. and the size of each ondexRangs) record 433 (8 bytes in 
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the preferred embodiment), the location of the first (fuein. 
loData ) fecord 436 is easily deduced. Normally a backup 
data file has only one andexRange> '*33 (i.e., a single con- 
tiguous range of file indices}, but it is possible after the 
Agent 108 has performed a consolidation operation 5 
(discussed below) for multiple non-contiguous ranges to 
exist in a single file. Each ondaxRanga) record 433 con- 
sists of two fields: an nndexBaso > '*34 and an (in(jaxCoLnt> 
435, each of which are 32-bit values in the prefened 
embodiment. The (indexBase) value 434 indicates the ,0 
first file index in the range. The (indaxCoum > value 435 
indicates the number of file indices in the range. The 
sum of the (MexCounV values 435 from all (indaxflanga) 

records 433 indicates the number of (fiieinioData) records 
435 in the file. In the prefened embodiment, the file ,5 
index associated with each (tiiemioOaia) record 436 is 
implicitly assigned sequentially from the ordered set of 
file index values generated by the (indoxRange ) record(s) 
433. 

In the preferred embodiment, each < fnamioData i ^° 
record 436 is of fixed-size, consisting of four 22-bit 
fields, as shown at 436 - 440. In particular, the tfiieintoPtr) 
value 437 points to the associated variable-length 
»iieinfo) record 408. The (fiiesiie) value 438 indicates the 
size of the associated file in bytes. The uMnteCRC > value 25 
439 is a hash value (a CRC in the preferred embodi- 
ment) computed over a portion of the directory entry for 
the associated file; useof this fixed-size value instead of 
a variable-length directory entry simplifies the search for 
matching files between users. The (paniaiFiiaCRC ) **0 is 30 
a hash value (a CRC in the preferred embodiment) com- 
puted over the first portion of the file. In the prefen-ed 
embodiment, it covers up to the first Np = 256K bytes of 
the file (which is all of the file in most cases). When 
searching for matching files across users, the backup 35 
application loads Np bytes of the file into memory and 
computes a hash value ((paniaiFiioCRC) ^). then per- 
forms a preliminary search through the global database 
(e.g.. 145) for matching <,i|g|n,o) records 408. If a match 
is found, then a more complete match can be verified 40 
using the full (fiieCRc > field 409. although there is usually 
no need to perform this further check since most files 
are smaller than Np bytes. Using this partial-file hash 
technique generally allows a single-pass search for files 
that are too large to fit into memory, instead of having « 
the read the entire file once to compute the jfiieCRC) 
and then a second time to back up the file contents if 
there is no match. 

There is one variable-sized ((j«info) record 408 for 
each fie included in the backup data file The (,itoCRc > so 
value 409 is a hash over the entire contents of the file: 
in the preferred embodiment, a CRC is used. The (bj,. 
Fields ) record 410 contains several small bit fields indi- 
cating various attributes of the (,ii,in,o) record. For 
example, the (rafcnii fiekJ 4n is a two-bitfield in the pre- ss 
ferred embodiment indicating how many external files 
are "referenced" in reconstructing the contents of the 
file, and can take on the values 0 (no external files), 1 
(one external file), or 2, while the value 3 is not allowed 



in the preferred embodiment. This particular limitation is 
imposed only to optimize the encoding of the ^^aiaPu 
field 419; in theory, there is no reason why more exter- 
nal files could not be referenced, although in practice it 
is very rare for more than one external file to be refer- 
enced; the previous file version, The value of the i,eK;ni 
field 411 indicates the number of meKer records 426 
that are included in the (fnemio- record 408. The (reiuvei • 
field 412 is a six-bit field in the preferred emoodiment 
and is defined to be the one plus the maximum (retuavei ■ 
value 412 for any external referenced file indicated in a 
(fiieRao fecond 426, or zero if (.e^no ^11 is zero. Thus, 
t*^e (retLoveF! value 412 counts the maximum levels of 
"indirection" required to access any portion of the file 
contents; this value is limited in the preferred embodi- 
ment to a user-settatile parameter N,, (typically in the 
range 5-10) in order to set an acceptalsle bound on 
access time to the contents of the file at restore time. 
Whenever the ,re(|.e„6i) value 412 would exceed the N^_ 
value if a particular external file were referenced, the 
data from the associated tikxk is duplicated instead of 
being incorporated by reference. The (sGbeaj) bit 413 
indicates whether the given file should be entered into 
the global directory database 145; it is , for new and 
updated files and 0 for all other files. 

"The (s«,kPis) record 414 comains a count (joenpt- 
cojnt) 'H5 (32-bits in the preferred embodiment) of the 
number of („okPoini) records 416 in the i^tkPtt) record. 
Each (seskPoim) record 416 consists of the starting (^^j. 
caiofte«i) value 417 associated with the (se«kFtointi 416, 
followed by a pointer ^^p^, 418 to the associated data 
"The <s9skPoini) array 416 is saved in sorted order based 
on the (logicaiottsaii values 417, allowing a quick binary 
search to find the (3«ikPoint: 416 tor any particular logi- 
cal offset in the file. The number of bytes of the file "cov- 
ered" by each (ssekPoini) 416 is easily calculated by 
subtracting its dogieatotisst) value 417 from the dogicaOH. 
se,; value 417 of the succeeding („BkPoint> 416 (or from 
«ie <fitesi2e> iieW 438 for the last <se,kPojnt) record 416). 
There is no firm limit in the prefen-ed embodiment for the 
minimum number of bytes covered by a (seekPoim :• 416 , 
but typically the blocks are fairly large (8K bytes or 
more), although this may decrease (or increase) as por- 
tions of external files are referenced. 

<dataPtr> ''^kJ 418 can take one of two forms: 
either a (daiaBiockPtri 419 reference to a (dataBioeX) 405 in 
this backLp data file, or a (exiamPtr) 420 to an external 
file. In the preferred embodiment, these two fiekJs each 
consist of 32 bits and are distinguished by the value of a 
single type bit in the ^„ora >■ as shown in 41 9 and 420. If 
<daiaPtr> ^'sW 418 is a (daia6io*Pir ) 419 (as deter- 
mined by the type bit being o as shown at 419). the f^^^. 
Piag) bit indicates whether or not the associated 
(daiaBbck; 405 is compressed. and the (b4o(*Offs) fieW 
421, which comprises the remaining 30 bits of the 
idaiaPtr) 418 in the preferred embodiment, points to a 
waiaBiock:. 405 in this backup data file. As discussed 
previously, each (daiasiock) 405 starts on a 4-byte 
boundary in the preferred embodiment, so that the 30 
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bits is sufficient to represent any (daiaBiock) o*feet in the 
file, little ua,apj.field4i8isa (^„emPv) (as determined 
by the type bit being , as shown at 420), the ;rafFiieNo> bit 
424 indicates which file is being referenced (hence only 
two files can be referenced in the preferred embodi- 5 
ment), and the (raions) ^^lue 423 is a signed relative log- 
ical offset from the dogicalOtlset) 417 of this (saeKPojn,, 

416, indicating the absolute logical offset in the external 
referenced file where the data associated with this (seek- 
Point) 416 can be found, Notice that accessing such an jc 
external block given this logical offset requires parsing 
the ifiieinio) section 408 and (swkPoint) records 416 of 
the referenced file in another t>ackup data file, which 
may in turn reference yet another external file; hence 
the limitation on the number of reference levels. In ,j 
the preferred embodiment, the .reions.' 'ie'd 430 is only 
30 tjits, so a referenced external block must start within 
+/- 512 (kbytes of the given dogicaotiMii -^l^, which is 
not a limitation in practical terms, although this restric- 
tion could easily be removed by extending the size of sc 
file idaiaPtr) ^'^^ ^1 8 when dealing with extremely large 
files. 

• The optional (faeRefi records 426 indicate which 
external file(s) are referenced by the (axtemPtr) "elds 420 

the (aeekPoint) ^"^Y 416. These <fiienen records 426 25 
are encrypted with the same encryption key used for the 
(dataaiock) records 405 for this file. The (fjei^, record of 
the ifiioRoi) 426 is identical in format to the «iiaiD > record 
214 used in the backup directory fi!e, containing the 
(fiisindax ) 21 5 and (u„rindex ) 21 6 fields that identify the 30 
particular file being referenced. The (docrypiKey) record 
427, which consists of 64-bits in the preferred embodi- 
ment, contains the private encryption key used for the 
referenced file. This key is also contained in the backup 
directory file(s) which contain (flieio, records indicating as 
this referenced file, but the key is duplicated here 
because it may only be othenwise available from a 
backup directory file of another user, which is encrypted 
with that user's personal encryption key. Hence, 
although the key is included here, 11 is encrypted to 
restrict access to only those users who have legitirrate 
access this file, as discussed below, so as not to com- 
promise the privacy of the referenced file. 

FIGURE 8 gives a detailed example of the (seekPts) 
record 41 4 for a hypothetical file X. The <seekF>tCount> of is 
this record Is S, as shown at 450. Thus, there are five 
(ssekPani) records, 451- 455. each of which is broken up 
into its (logicaiofiMt) field (e.g., 456) and its <aauF>ir> tield 
(e.g., 457, 458, 459). The first <se,KPoim) record 451 has 
a starling logical offset of 0 as shown at 456. and tfis so 
faeekPoini) fecord covers the first 8192 bytes (0-8191) of 
file X, since the second (jeeKPoim* 452 starts with logical 
offset 8192. These 8192 bytes associated with the first 
(wekPoint) record 451 are found in a (dstaBiock> within 
this backup data file, as is indicated by the type bit 0 at 55 
458 which identifies the (daiaPtr) record of 451 as a < 

dalaBkxkPtr)- (b(od<Otte) tieW 457 Of the first (seekPoim) 

record 461 contains the value 128, indicating that the 
associated <dataBio*) "s to be found at offset 512 (i.e.. 



4'128) in this backup data file, and the , bit in the .^^g^,. 
Flag, field 459 indicates that this (aaiasiock is com- 
pressed. Similarly, the second (sookPomi) record 452 
covers the bytes 8192-1 1999 of file X, but these 3808 
s bytes are to be found starting at logical offset 8492 of 
the external file indicated by the first (fueqa,. record (ret 
file »0) in this (fueinto) record. The offset 8492 is com- 
puted by adding the (iogicaioHset> value of the second 
teeekPoint) record 452 (i.e., 8192) to the (,eiott5 ■ value of 
ic the (dataPir) record of the second issskPwnt > record 452, 
which is an fenemPtr) as indicated by the , type bit in the 
(dataPtr lithe „e,p 

ileNo> 'ield of 452 indicates which .voipnei 
is referenced (0 in this case). The third (saekPoint) record 
453 covers bytes 12000 - 16383 of file X and indicates 
15 an unconpressed data block starting at offset 4080 of 
this backup data file. The fourth (seekRam > record 454 
covers bytes 16384 - 23008 of file X, and these 6625 
bytes are to be found in reference file #1 at logical offset 
15384 (ocgicalOlfsel. + (relOHs) = 16384 - 1000 = 15384); 
sc note that (reiotis) is a negative number in this case. The 
fifth (and last) (^ekWnt) record 455 covers all the 
remaining bytes of file X; for example, if the (fiias.ie) is 
30000. this block consists of the 6991 bytes 23009 - 
29999 Those bytes are found in a compressed ,d.uB- 
25 lock) at offset 8472 of this backup file. This example 
shows how simple it is to interpret the feoekPis > structure, 
and it is obvious that a binary search on the aogicaottsei ) 
field can be used to locate any section(s) of the file very 
quickly 

30 The ,^,rjnis) section 428 of the (rM„^) record 408 
contains hash functions or "fingerprints" computed over 
fixed-size portions ("chunks") of the file contents. The 
purpose of these fingerprints is to allow efficient proba- 
bilistic searching of matching chunks between file ver- 
35 sions without having to fully extract the contents of the 
previous file version. The idea of using fingerprint func- 
tions in this fashion was first conceived by Karp & Rabin 
IKarp. Richard M.. and Michael O. Rabin. "Efficient Ran- 
domized pattern-Matching Algorithms". Harvard Univer- 
■M sity Center tor Research in Computing Technology. TR- 
31-81, December 1981]. Fingerprints are particularly 
effective »*ien performing backup of modified files over 
a low-speed communicafions link when the backup data 
files are at the remote she, as discussed in a subse- 
ts quent section. In the preferred embodiment, although 
there is no absolute need to use the fingerprints (since 
the pra/ious file contents can be explicitfy produced for 
chunk matching), the fingerprints are stored in the 
backup data file anyway to facilitate such bandwidth 
so optimizations; in particular, even over local area net- 
works it nnay be desirable to minimize network traffic 
when backing up large files with only small modifica- 
tions. The size of the chunk used for fingerprinting, 
which may vary from file to file, is indicated by upchuik- 
5* siM) 429 and is typically in the range of 256 to 8192 
bytes; a value of 0 for <,pChunkSi2« > 429 indicates that no 
fingerprints are stored. Uke the (d«ryptK.y) record(s) 
427, in the preferred embodiment, the (^jntns) record 
428 is encrypted using the associated encryption key 
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from fhe "fileDecryptKey" section of ttie backup direc- 
tory file. 

The basic idea behind fingerprinting, as described 
in detail by Karp & Rabin, is to choose a hash function 
which is easy to "slide" over a chunk of data. In other 
words, as the chunk starting location is moved from one 
position in the file to the next, the "oldest" byte exits the 
chunk "window", the intermediate bytes shrtt over one 
location, and a new one enters the window. Karp & 
Rabin describe several types of linear fingerprint func- 
■ tions which are easy to update given the current finger- 
print value and the oldest and newest bytes. For 
example, a modulo 256 sum Is a particularly simple 
case (too simple to be useful in practice), but CRCs and 
other similar functions are quite acceptatde. Given the 
set of fingerprints for the chunks of the previous file con- 
tents, the fingerprint function is computed by sliding 
over chunks of the current file coments, checking for a 
match with any of the previous file fingerprint values at 
each byte location. When a match is found, that chunk 
in the cun-ent file is assumed to match the chunk asso- 
ciated with the fingerprint value in the previous file. The 
. fingerprint function can be chosen to be large enough 
(72 bits in the preferred embodiment) that the probabil- 
ity of false match (e g , 2"^^ (y approximately 10"^) is 
smaller than the probability of storage medium failure 
(typically IQ-'^ so that no further validation is neces- 
sary. Alternately this sliding fingerprint mechanism can 
be used solely as a search technique to identify areas of 
probable matches and then fully validate them by 
extractng the old file contents and Performing a com- 
plete compare. It is also possible to use fingerprints only 
in a non-sliding fashion; this approach works particu- 
larly well for very large (e.g., database) files where 
records tend not to move, while for smaller files, where 
the bandwidth consumption is not as much of an issue, 
a full compare could be performed in this case. In an 
alternate embodiment, a global database could be built 
of chunk fingerprints instead of entire files, allowing 
matching of portions of files across users, but the 
expected gain in storage space from such a scheme 
does not appear to be worth the extra overhead 
required. 

in the preferred embodiment, each jngerpnnt) record 
430 consists of nine bytes (72 bits) of fingerprint func- 
tion (CRCs), plus the first three bytes of the associated 
chunk, for a total of twelve bytes. Using these extra 
three bytes allows the fingerprints to be computed and 
compared on a sliding dword-by-dword basis instead of 
a byte*y-byte basis, which speeds up the conputation 
considerably However other than speed, the net result 
Is the same as a byte-by-byte sliding fingerprint conpar- 
ison. There is one (fingerprint) record 430 per chunk of 
the file; however, in order to save disk space, no „ir^,_ 
prini) records 430 are included for chunks which are : 
entirely contained in external file references (via 
textsmPtr) records 420) wrth identical (,pchunksi«> values 
429 and which are on chunk boundaries in the refer- 
enced file, since the fingerprints for those chunks are 



already contained in the oneinio 408 for the referenced 
file. 

There are many possitile variations on the particu- 
lar layout of records in the backup data file of the pre- 

5 ferred embodiment. For example, in an alternate 
embodiment each <,,isinto' record 408 could be placed 
directly after the set or .^aaBiock records 405 associ- 
ated with the file instead of in a separate section: for 
example, this change might be represented by simply 

'0 changing definition 400 to read: 

(bt<upOatflFils> ■'== (header i[(dataBlocl>' ' (filelnkj j" ;fin. 

Similarly, some fields, such as ;fjieCRc ; ''09 and (jsGicbai ■ 
413, coUd be moved from 408 to i,M„,oPtr^. 

15 432, or vice-versa. In some file systems (e.g., Windows 
NT NTFS), 64-bit file pointers would be used instead of 
the 32-bit poimers of the preferred emt>odiment. It 
vrould also be simple to modify the format slightly to 
allow for more reference files or more reference levels. 
Such changes do not affect the basic iidea, and the par- 
ticular record formats of preferred entxxJiment 
descrbed here are not intended to limit the scope of the 
present invention. 

1 .3 Global Directory Database RIe 

With a knowledge of the information contained in 
the backup data and directory files and of how the infor- 
mation is used to represent file data contents, the tech- 
nique for searching for matching files can easily be 
explained. In designing the global database, it was 
assumed that there could be millions (or tens of mil- 
lions) of new/updated files entered into the database. 
For example, a survey of ninety user workstations at 
Stac (the assignee of the present invention) revealed a 
total of about 250.000 unique files across all the disks, 
and the preferred embodiment is designed to handle 
systems with at least that many backup nodes. Thus, it 
is inrportant to minimize the network bandwidth con- 
sumed by the search process, which might easily dwarf 
the file data traffic during backup unless great care is 
taken in the database design. In particular, several con- 
ventional datatHse approaches (e.g., B-Tree) were con- 
sidered and rejected in light of this concern. While there 
may be other types of database architectures that work 
well, the structure of the database of the preferred 
embodiment is particularly effkaent for the type of 
searches required here. 

During the backup process, each node may have 
thousands of new/updated files that need to be 
searched against the global database. Generally, there 
will be considerably fewer such files once fhe initial 
backup is completed, but the worst case must be han- 
dled. By contrast, there may be millions of files already 
entered into the database. Thus, it seems initially that a 
clientyserver embodiment with a backup server, in which 
the client sends its (relatively small) list of new/ipdated 
files to the server, which in turn does the matching 
against fhe large global database, should have a signif- 
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icant advantage in network bandwidtti usage over a 
shared-file system. Howe/er, the overhead of perform- 
ing the search in the shared-file environment of the pre- 
ferred embodiment is optimized to the point that this 
drawback is not significantin practice. 5 

When searching for matching files across users, it 
is usually deemed sufficient to have matching file size, 
file name, time/date, and hash value (e.g.. CRC) com- 
puted over the file contents. While this approach does 
involve a finite (though minute) probability of false rc 
match, the error probability is acceptably small tor 
almost all practical applications. In an optional user- 
invoked "exhaustive compare" mode, this probabilistic 
type of match serves only to initiate a complete byte-by- 
byte conparison of the contents of the two files; how- ts 
ever, the overhead of this mode is large enough, partic- 
ularly in light of the practically negligible improvement in 
the level of certainty obtained thereby, that Invoking 
such "skeptical" behavior Is best done infrequently, if at 
all. In alternate emtiodiments, the match criteria can be sc 
further loosened not to require a matching file name or 
time/date; for example, two files REPORT.DOC and 
•REPORT.BAK' might be judged to be matches if all 
other parameters are equal There are many variations 
on this theme; for instance, pemaps just the file names, 25 
excluding extensions, are compared, or perhaps only 
the first few (e.g.. 4-6) characters of the file name are 
compared in an attempt to include minor file renaming 
changes, such as 'REPORT to REPORTr. In general, 
however, the file size (or at least some number of least so 
significant bits of the size) and the hash value on the f i e 
contents are required to be equal in order for a fie 
already in the database to be judged identical to a 
new/updated file being backed up. In the preferred 
embodiment, in order to work around the "problem" of 35 
the variatjle length of the file name (or of other directory 
attributes) in formatting the global database entries, a 
32-bit hash (actually, a CRC, (diri^DCRC) 439) over the 
relevant directory entry information (e.g., file name, 
time, date, and size) is used for comparison instead of -w 
the full directory entry. In addition, the complete hash 
value (a 32-bit CRC in the preferred embodiment) over 
the file contents is compared, as well as the least signif- 
icant 1 6-bits of the file size, ft all of these values match, 
the file being backed up is considered to be a match to 4S 
the file in the datalsase, resulting in an false match prob- 
ability of less than 2'^ (lO'^*). Clearly the amount of 
matching required can be tailored to the specific error 
probability acceptable for any given environment (e.g., 
bjf increasing the size of the CRCs), and such changes sc 
would still fall within the scope of the present invention. 

It is useful to note at this point that these various 
levels of matching fOes aaoss users in the global data- 
base are all more rigorous in general than the level of 
effort used to identity unchanged files from the previous ss 
backup of the same user. In the preferred enixxiiment, 
as is quite common in backup applications, the default 
behavior is to consider a file unchanged if its file size, 
time, date, and name are unmodified from the previous 



backup. As discussed above, rt is always possible to 
pertorm. at the user s option, either an exhaustive com- 
parison of the contents of the apparently unchanged 
files or a comparison of CRCs on file contents, but the 
improvement in certainty level is rarely considered to be 
worth the extra effort and ovemead. 

Given the winmoCRC- 439, ^11,5^0. 438. and «i,ecRC - 
409 values for a particular file to be backed up, a search 
through the glotsal database of the preferred embodi- 
ment is performed in an attempt to find a matching 
entry As noted previously, the (parnaiFiisCRC 440 value 
is actually used initially instead of the full diieCHC 409 
as an optimization, since the former covers the entire 
file in most cases; for large files the (ueCBC > 409 is then 
verified in the preferred embodiment by looking into the 
backup data file containing the (Hiemio) 408 for that file. 
Each global database entry contains the airintoCRC > 439 
(tour bytes), (fjiesize) 438 (actually only the least signifi- 
cant sixteen bits in the preferred embodiment), and (pa,. 
naniioCRC! 440 (four bytes) values for the associated file, 
which are extracted from the backup data file containing 
the (fiieintoDaia) record 435 for that file. In addition, each 
entry contains the (,h,id) record 214 (six bytes) which 
can be used to locate the actual file data contents. The 
total size for an unconpressed database entry is thus 
fixed at 16 bytes in the preferred embodiment. If there 
are N = one million files in the datatiase, downloading 
an entire global database from the backup storage 
means 101 in order to perform the search would reqiire 
a download of 16 MB of data, if no effort were made to 
minimize this overhead. While this amount is considera- 
bly less ttian what would be required to download a 
database full of complete directory entries (e.g.. with 
entire file names), it is still far too large for an environ- 
ment where dozens or hundreds of nodes on the net- 
work may be performing backup. In an alternate 
embodimem, the conplete maSaau (liieCRO. and other 
fields could also be stored in each global database 
entry, slightly reducing both the probability of false 
match and the search time, at a small cost in the size of 
the global database file 145, but such improvements are 
minor at best in practical terms. 

To minimize the data transfer overhead and the 
search time associated with the global directory data- 
base 145, it is organized into two levels as shown in 
FIGURE 9, taking advantage of the effective randomiza- 
tion of search values due to the nature of a CRC func- 
tion, as used in (dtMntoCRo 439 and ipaniBiFiteCRc > 440. 
Each entry of the first level 500, which is actually repre- 
sented in two struaures 502 and 505, contains only a 
subset of the bits of the <dHn.oCRc, 439 and (p„,i.|R^. 
CRC ) 440 fields. Each entry in the second Iwel 501 con- 
tains the remaining tiits needed to constitute the entire 
global database entry (e.g., 508, 509, 510. and 511). 
and each entry also includes a 1 6-bit C RC over the sec- 
ond-level entry to allow for a con^uption check. TTie 
number of bits included in the first level 500 is fixed in 
each global datatiase file, although the actual nunt>er 
will increase in general as the number of database 
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entries grows. The first level 500 is downloaded to the 
node from the backup storage means 101 by the 
backup program, and its contents (502 and 505) are 
used as a quick fitter to limit inquiries into the (much 
larger) second level 501 to only those entries which 5 
have a very high probability of being a match. In the pre- 
ferred embodiment, to minimize download time, all 
entries in the first level 500 are packed at a bit level and 
are unpacked after downloading, while entries in the 
second level are byte-aligned for simplicity. ,0 

The entries in both levels are stored in the same 
order, sorted by the value of (dirinfoCRO 439, so that 
given the index of an entry (e.g., 512) in the first level 
500, the position of the corresponding entry {e.g.. 513) 
in the second level 501 can be easily computed. In other is 
words, the «h entry in the first level 500 corresponds 
directly to the Mh entry in the second level 50 1 . The first 
level entries are actually stored in a compressed form to 
save download time, using a counts array 502 and par- 
tial entry array 505. This simple compression is so 
achieved by noting that, since the entries are sorted by 
the value of (dirmtoCRO 439. the leading (most signifi- 
. cant) bits of consecutive airinfoCRc > values 439 will tend 
to be equals. Thus, instead of storing these leading bits 
for each entry, a counts array table 502 of fi/ = 2'"° 2S 
entries is included, where the value of mo is selected by 
the Agent 108 as discussed t>elow. The /th array entry, 
ny, containing the number of consecutive (drinbCRO 
entries 439 with the leading mo bits having a value of i. 
as shOKvn at 504. For exanple, in FIGURE 9, no is 4, 30 
covering the first four entries in the tables 505 and 501. 
for which the leading mo bits of (dirmtoCRo 439 are 0: 
similarly n, is 3, covering the next three datatiase 
entries, for which the leading mo bits of MinmoCRC) 439, 
interpreted as an integer, are 1. When the database is 3S 
created, the Agent 108 chooses the value of mo based 
on the total number of global director; entries (N) in the 
file; a typical value is mo = 16 for N larger than 64K. 
Note that N = I ny , where the sum is over all values of / 
= 0 .. M-1 . Since the values of <dirintoCRC :■ 439 in the file 40 
are effectively randomly distributed, the n, values fave a 
distritjution with mean N/M and a fairly small range. 
Thus, to minimize storage space further in the prefened 
embodiment, instead of storing the actual values n. . 
these values are represented in the counts array 502 by 45 
"y ■ "min. where n^n is the minimum over all n^ values. 
Each count can then be represented in 

s = flogjd + rv^- 1 

so 

bits, where n^,, is the maximum over all ny values. The 
values n„jn and s are computed by the Agent 108 when 
the global datatase file is created and are stored in the 
header of the global database file. In an alternate ss 
embodiment, it may be possible to reduce the size of the 
counts array 502 even further using a Huffman or arith- 
metic code, but such gains would be minor because the 
counts array 502 constitutes only a small part of the size 



of the first level 500. 

A concrete example is the easiest way to clarify this 
simple encoding. Suppose that we have a total of N = 
one million database entries. If we choose mg = 1 6, then 
M = 64K, and the average value in the counters array 
502 is N/M = 16. Suppose that we then find n^.^ = 2 and 
"max = 30. Then s = 5 bits, so each count entry n^ is rep- 
resented in five (packed) bits by the value n, - 2, for a 
total of 40K bytes (64K entries at 5 bits each). Without 
using a count array, each database entry in 502 would 
have contained all mo = 16 leading bits of the MirintoCRC ■ 
value 439. tor a total of nearly two megabytes (1953K 
bytes), so using the count array in this case saves a total 
of nearly 191 3K bytes In the size of the first level 500. 
Note that the amount of savings is not very dependent 
on the value of s; using simulations on a random distri- 
bution, it has been obseni-ed that even for N/M as large 
as 1024, which conesponds to 64 million files in the 
database if mo = 16, well over 99.9% of all count anay 
distributions can be represented by s = 8 bits or legs. In 
practice, the Agent 108 tries various values of mg to 
minimize the size of the first level 500, although it 
appears empirically that the amount of savings is not 
terribly sensitive to the choice of mo as long as it is close 
to the mo value that produces the minimum; in other 
vrards. simply using mo = 16 appears to work fairly well 
in most cages of interest 

With the counts an'ay 502 used to represent the first 
mo bits of the' <dirin(oCRc> value 439 very eflidently, the 
remainder of the first level 500 consists of an array 505 
of N entries, packed at a bit level. Each entry contains x 
bits of the uirinioCRC ) value 439 (beyond the most signif- 
icant mo bits) and y bits of the (p„,«iFii,cRC > 440. The 
values X and y are chosen by the Agent 1 08 (and stored 
in the global database file header) based on mo and the 
total number of emries N in the global database 145. 
Since the entire first level 500 is downloaded, the idea is 
to trade off the size of the an-ay 505 to minimize the 
number of accesses required into the second level 502 
to validate a match. For example, using N and M from 
the above example, if we choose r = 10 bits and y = 0 
brts, then the table 505 consists of a total of atxjut 
1220K bytes (one million entries at 10 bits each); the 
entire first level 500 consists of at»ul 1260 K bytes 
(1220K + 40K), as opposed to the 16lvl bytes required 
for a complete download of the entire database. Since 
we have mo + x = 26 bits of (drmioCRCi 439 thus repre- 
sented by the first level 500, the average probability p, of 
a false second-level match based on a first-level match 
is then given roughly by p, = HIZ^^ = 1/64. assuming (as 
we are) a random distribution of (dirmtoCRCi values 439 
in the database. In other words, when filtering database 
inquiries at the first level, about 63 of 64 inquiries that 
match at the first level will result in matches at the sec- 
ond level also, in this example. Since every inquiry into 
the second level 501 involves a disk access into an 
entry (e.g., 513) containing the remaining fields of the 
global database entry, it is important to minimize spuri- 
ous accesses. Typically a value of p, in the range 1/16 to 
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1/256 gives a reasonable tradeoff between search per- 
formance and download size For example, if we 
increase x to 1 1 bits in this example, we decrease to pj 
= V128 at a cost of about 1 25K bytes in the size of the 
first level. Although y = 0 in this example, the / bits of 5 
(partiaiFiteCRC) csf^ be used to extend the tradeoff 
range as N becomes very large, or in the unusual case 
where many files with the same name«ime/date/size 
(i e- (dirinfoCRC) '*39) exist with different file contents 
(arxJ thus (partiaiRieCRO values 440). The Agent 108 7c 
determines all these parameters at database creation 
time based on the statistics of the entries in the data- 
base.' In the preferred embodiment, mo + x- is always at 
least 16, meaning that the first level entries contain at 
least the 1 6 most significant bits of the MinnioCRC ) value ,5 
439, so that only the least significant 16 bits of <cjjrin. 
locRC) 439 need to be kept in the second level 501 at 
508. 

At the beginning of the backup process, the backup 
software of the preferred embodiment loads jmo mem- so 
ory the first level 500 of the global directory database 
file 145, either from the backtp storage means 101 or. 
to. minimize network bandwidth consumption, from 
cached copy in a directory on a disk local to the node. 
For each new/updated file to be backed up. a search is 25 
performed through the first level database entries to gee 
if there is a match. If no match is found at this level (the 
"no match" case), there is no matching file anywhere In 
the database, so the backup proceeds to copy the fie 
contents into the tjackup data file, which may involve ao 
computng differences from the previous file version in 
the case of an updated file. If a match is found at the first 
level, the corresponding second-level entry (or entries) 
is retrieved and compared; if no match is found here, the 
backup prcx;eeds as in the "no match" case just dis- 55 
cussed. The position of the con-esponding second-level 
entry is easily determined, as discussed above, 
because its ordinal location in the second level is the 
same as the ordinal the associated first-level entry If a 
match is found at the second level, further inquiry into 40 
the backup data file containing the (UMnh) and 
Data, records 408, 436 associated with the file may be 
necessary In some cases, depending on the size of the 
file (e.g., aiieCRO be needed for large files) and 

whether the user has enabled the 'exhaustive compare" 4s 
mode, but in most cases a match to the global directory 
entry at the second level is sufficient to indicate a fie 
match. If it Is ultimately determined that a complete 
rratch has occun^ed, the („teiQ, 214 included in the 
<riieEniry> record 207 Of the backup directory file for ttis so 
backup is set to indicate the matching file in the global 
database, so no file data needs to be saved in the 
backup data file for this backup, and there is no new 
«iiaind9,> 215 assigned, nor a <ti(ointe) sectkan 408 
added. 

The particular first-level search mechanism used in 
the prefen-ed embodiment is very simple, and there are 
many other well known search techniques that could be 
used. The key point here is that, after the first-level data 
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is downloaded, it is all available locally at the node, so 
there is never a need to access the remote backup stor- 
age means 101 to identify first-level matches. In the pre- 
fen-ed embodiment, it is assumed that the entire first- 
level data can fit into main memory during the search 
process; were this not the case, a virtualized (disk- 
based) search could be designed, using well known 
algorithms, that might be considerat3ly slower but would 
still achieve the same result. The preferred embocSment 
builds two arrays in memory as shown in FIGURE 10. 
The main array 525 has N entries, each containing the 
X -(■ mo bits (527) of (dirinioCRC ) 439 and >- bits (328) of 
ipartiaiFiioCRC ) 440 from the first-level entry sorted in the 
same order as In the first level of the global database 
file. In other words, the array 525 is effectively a memory 
Image of the contents of 505. The pointer array 520 con- 
sists of T = 2""^ entries, whore m, is a number of bits 
chosen based on the total nurrtier of global database 
emries N and the amount of memory availaljle in order 
to optimize the search process; note that m, may or 
may not be equal to mo- Each entry in the pointer an-ay 
520 contains a pointer P^. info the main an^ay 526. The 
index into the pointer array 520 is computed by extract- 
ing the most significant m, bits of the (di,i„foCRC) value 
439 for the lie in question. For example, Pq 521 points 
to the first entry in 52S, while P, 522 points to the fifth 
entry in 526, which in this example is the first entry for 
which the m, bits of ujrinfoCRc ) 439 in question have the 
integer value 1 . Similarly, 523 points to the first entry 
in 526 for which the m, bits of (dirmioCRO 439 in ques- 
tion have the integer value k. The count of entries in 526 
to be searched for each index k is easily obtained from 
the difference between successive pointer entries, P);.^, 
- Pk : an extra "dummy" entry Pj 525. which points just 
past the end of the main array, is appended at the erxl 
of the pointer may 520 in the preferred embodiment so 
that the same ooum computation can be performed for 
the last entry Pj., 524, without requiring any special 
case logic. 

In the preferred embodiment, entries to be added to 
the global directory database file 1 45 are extracted from 
the backup data files (e.g., 144) by the Agent process 
1 08 as part of the migration of the backi^) data files from 
the \BACKUP\USER path (e.g., 125) to the 
\BACKUP\SYSTEM path (e.g., 129). The Agent 108 
first verifies the CRC covering the (,ii,inioOat.) entries 
436 in the backup data file to guarantee that no cor- 
rupted entries are added to the global directory data- 
base. A new global database file 145 may then be 
CTeated. consisting of the old entries merged with the 
new entries. In the preferred embodiment, the new data- 
base file is Initially created by the Agent 108 under a 
temporary name so that backup processes may con- 
tinue to use the cun-ent database file. Once creation of 
the new file is conpleted. its name is changed to a valid 
global directory database file name which will then be 
accessed by siiDsequent backup operations. In the pre- 
fen-ed embodiment, the name of global directory data- 
base files have the form GDnnnnnn.GDD. where 
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nnnnnn is a number which is iraemented each time a 
new global directory database file is added. For exam- 
ple, the first file would be GDOOOOOl .GDD, the second 
would be GD000002.GDD, etc. Only a small number 
(typically 1 - 4) of the most recent versions of such files s 
IS retained; older versions are deleted once they are no 
longer in use. Thus, for example, after some time there 
might be two tiles GD000138.QDD and 
GD000139.GDD stored in the \BACKUP\SYS- 
TEM\GLOBAL directory 1 27; each time a backup oper- u 
ation begins, the backup process will select the "latest 
version of the global directory database file 145 availa- 
ble (GD000139.GDD in this example). 

This method of completely rewriting the database 
allows the optimized search structure discussed above rs 
to be maintained, as opposed to a conventional data- 
base design (e.g., using B-Tree structures to add new 
entries). Fortunately, the "batch" mode of operation 
inherent in backup makes such an approach acceptable 
in this application. However, once the backup system 2c 
has been in use for a while, the number of additional 
entries io the global database for each new backup 
often becomes a very small fraction of he overall data- 
base size, particularly since only new and updated files 
are added to the database. For example, there might be 2S 
one million entries in the global database, bul a new 
backup process might add only a few dozen new 
enfries. In this case, rewriting the entire global database 
can be an extremely slow process, and downloading the 
new database after each backup coukj also be slow To 3o 
minimize such overhead, in the preferred embodiment 
the Agent process 108 may post "update" directory files 
147 in the \BACKUP\SYSTEM\GLOBAL directory 127 
These update files 147, which are basically identical in 
structure to the main global database 1 45. contain only 35 
ttie new entries to be added to the global database 
Smce some of these update files may be quite small 
the Agent 108 may choose to store them in a simplified 
format with mo = 0, x = 1 6, and / = 0. so that there is no 
count table 502. 

In ttie preferred embodiment, each update direc- 
tory file IS given a file name which links it to the associ- 
ated "base- global database file; the riming convention 
IS GUxxxnnn.GDU. where nnn is the last three digits of 
the base global database file name, and xxx is the « 
update number. For example, the file GU003138GDU 
would be the third update to the base file 
GD000138.GDD. Since only,a few global database files 
are retained at any time, the three digits nnn are always 
sufficient in the preferred embodiment to identify the so 
associated global database file unambiguously 

The backup software usually maintains a simple 
cache on a local disk of the last global/update directory 
fileCs) downloaded from the backup storage means 101 
so rt can speed up the database first-level download 
process. In the preferred embodiment, each update file 
contains all the updates (i.e., a differential update) to its 
associated main database file, so the backup process 
only has to download the most recent update file at any 



time. In an alternate embodiment, the updates could 
instead be incremental in nature so that all update files 
would have to be downloaded, or both incremental and 
differential updates could be stored, giving more optimi- 
zation flexibility to the backup software local cache 
logic, Once the update list reaches a certain size (eg 
10 percent of the size of the global oatabase) or a cer- 
tain number of update files {e.g. , 500) have been added 
the Agent 108 rebuilds an entirely new global database 
file 145 containing all entries in the main and update 
database file(s). These particular settings governing 
how often a new global database file is built are control- 
led by the backup system administrator on the Agent 
node 107. Even after building a new global directory 
database, the Agent 108 may leave the oW onefs) 
around for a while and may even continue adding 
updates to the old file(s), so that the local cache logic 01 
the backup software may optimize its download strat- 
egy In general, only infrequently does the backup soft- 
ware need to download the first level 500 of an entire 
global directory database file 145 from the backup stor- 
age means 101, thus minimizing the startip time for 
each backup operation. 

1 .4. Other Backip Files 



FIGURE 3 shows several file types other than those 
discussed above, tvlost of these files are either redun- 
dant (i.e., can be regenerated from other files) or are 
ancillary at best to the present invention. A brief 
descrption of the contents and uses of these files is 
given here for completeness. 

As discussed previously, an Index Range Lookup 
iile (e.g., 151) is built and maintained for each user by 
the Agent process 108. This file is constructed from the 
contents of the migrated backup data and backup direc- 
tory files (e.g.. 148, 149). it includes a table indcating 
the direaory/file index ranges of each backup directory 
and backup data file, respectively. This file is thus 
' emirely redundant and can be thought of as a table of 
contents for the backup directory/data files. Its contents 
are organized to allow a quick binary search to deter- 
mine while file contains a given directory/file index for 
the user, instead of having to open each file in tum to 
perform such a search. This file is not encrypted 

The Backup Log file (e,g„ 150) is also a redundant 
file, built and maintained by the Agent 1 08 for each user 
It contains a copy of the "bkupDescripBon- section of 
each of the user's backup directory files. This log file is 
typically used at restore time to present a list of availa- 
ble backups to the user, including the annotation string 
provided by the user when the backup occurred With- 
out this file, the restore software would have to open 
many backup directory files to present such a list which 
could be quite slow. The contents of this file are 
encrypted using the same encryption key applied to the 
backup directory ties. 

The User Account Database file 1 46 is maintained 
by the backup administrator software. It contains the 
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2.1 Keeping private files private 

The present invention includes a simple and novel 
technique that uses encryption to restrict access to a 
user's data on a file-by-file basis; in particular, only 
those users who in fact have (or once had) a valid copy 
of a file may reference that particular file. The data of 
each file in the backup set is stored in an enaypled form 
in the backup data file, where the encryption key is 
based on a fingerprint (e.g., CRC) of the file's data itself. 
These "encryption" fingerprints themselves are then 
stored in the 'lileDecryptKey" section of the tiackup 
directory file, which is itself encrypted with a key that is 
accessible to the user only by supplying a password. In 
addition, as discussed previously. (jnBRao records 426 in 
a backup data file also contain the (d«crypiKey) record 
427 for a referenced file, but these records are also 
encrypted, using the encryption fingerprint of the refer- 
encing file, to prevent Indirect" access. Thus, users can 
only successfully decrypt a file's data if they have the 
correct encryption fingerprint, presumably obtained by 
computing the f ingerpri nt over their own copy of the file. 
In this way, a user has access only to encrypted ver- 
sions of the private files of other users, while at the 
same time having the ability to deaypt files which are 
common (and thus not private). In the preferred embod- 
iment, each 64-brt encryption fingerprint is algebraically 
independent of the (fjngsrPmti values 430 and is the 
combination of a CRC and a single non-linear check- 
sum function over the first 256K bytes of the file con- 
tents. One interesting property of this scheme is that, if 
the "original" owner of a file forgets his password, he will 
effectively be denied access to his files, while other 
users can continue to get access to the ties they share. 

It is also true that a user may wish to keep private 
the mere existence of a certain file or a name of a file. 
Thus, in the preferred embodiment, the <voiumoDirinte) 
section 200 of each backup directory file (e.g., 143) is 
also encrypted with the same user-specific key as is 
used tor the "fileDecryptKey" section. Enough informa- 
tion is stored in the backup data file to allow a separate 
user to gain access to the (encrypted) data portion of a 
given file without knowing the name of the file, so unin- 
vited users are not able to "peruse" the directoryrtile 
tree of another user without knowing his password. 

2.2 "Backdoor" Data Access Policy 

The need for data privacy must be balanced in the 
corporate environment with the right of the company to 
retain access to its intellectual property (e.g.. computer 
files) in the case of an employee who is unwilling or una- 
ble to produce his password. Such cases coJd easily 
arise when an employee forgets his password, becomes 
disgruntled, or is the victim of a disabling or fatal acci- 
dent. Typically, the cuaent version of the user's data 
would be available directly from the workstation disk, 
but there are clearly scenarios where it is critical for the 
corporation to access the user's backup data sets. 



It was initially envisioned that the backup software 
should allow the user alone to set his password, without 
any administrative "back door" to the backup set data. 
However, it was decided that such an approach does 

5 not give the corporation the ability to recover its data in 
any of trie "disaster" scenarios mentioned above. Fur- 
ther, it became quickly aoparent that there were certain 
operations, such as changing passwords and consoli- 
dating backup sets, which became very dfficult or 

10 impossitjie for the Agent process 108 to perform in such 
an environment. Thus, the preferred embodiment is 
designed to maintain very high security and privacy 
between users, but the backup administrator does have 
the ability to access a user's backup data if necessary. 

ts For a user who insists on maintaining ultimate pri- 
vacy of certain personal files, there are several options, 
although some (or all) of these options may be unac- 
ceptable to his manager. First, he may opt not to use the 
backup software of the present invention. Second, he 

so may encrypt the files in question on his local disk using 
a separate enayption utility Third, he may exclude the 
files in question from the backup set. In the preferred 
embodiment, the administrator has access to each User 
Preference file (eg.. 142), which contains the 

25 exclude/include list so that an audit may be conducted 
by management of those files and directories which are 
not being backed up. 

2.3 Encryption Key Protocols 

30 

In any system using encryption, careful attention 
must be paid to how the encryption keys are handled, 
and the present invention is no exception to this rule. 
This section discusses the generation ar>d use of 

35 encryption keys in the preferred embodiment in order to 
insure privacy 

As shown in FIGURE 11, when a new user account 
Is added by the administrator to the backup system of 
the preferred en*odimeni. the administrator software 

40 generates a user-specific random unique enaypton 
key (userDirKey 541) that will be used to enaypt the 
user's backip directory files. As we hiave seen, the "file- 
DecryptKey" section of these directory files contains the 
keys 543 (generated from fingerprint functions on the 

4S file contents) used to encrypt the file data 542 in the 
backup data file. The userDirKey 541 is placed in the 
User Account Database file 146, where it is encrypted 
according to a user-supplied password 540. This pass- 
word 540 may be initially supplied by the user to the 

50 administrator, or it may be chosen by the administrator 
and given to the user (normally with instructions to 
change it upon first use). 

The administrator software of the preferred emtDod- 
iment also stores the user password 540 and the user- 

55 DirKey 541 in a separate section of the User Account 
Datatase file 1 46 which is encrypted using an adminis- 
trator password. Actually, what is stored is the enayp- 
tion key (a message digest in the preferred 
embodiment) generated from the password, not the 
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password itself. Thus, the administrator has a "back- 
door" patti to decrypt the user's directory files if neces- 
sary In addition, the administrator may configure the 
Agent process 108 to change the userDirKey value 541 
from time to lime and re-enaypt all user directory files 5 
to guard against the possibility that a hacker has some- 
how obtained access to the user's password 540 and/or 
userDirKey 541 . Although such a change may require 
some time to complete, in the preferred embodiment the 
backup directory files for a user remain "on line" during ^c 
this operation. This is actually accomplished by storing 
two userDirKey values (a cun-ent and an "old" value) in 
the account entry tor each user in the User Account 
Database file 146. If a decryption checksum fails using 
the current userDirKey value, the Isackup software of the is 
prefen-ed embodiment automatically tries the Old user- 
DirKey value instead. Thus, the Agent 10B first sets the 
old userDirKey value to be the current userDirKey value, 
then sets the new current userDirKey, and finally pro- 
ceeds to re-encrypt ail the backup directory files. At any so 
time during the re-encryption process, one of the two 
keys will work. 

■ Tlie user may change his password at any time by 
posting a password change request in his Password 
Log file (e.g., 140). This request is encrypted with the « 
cun-ent userDirKey value 541 and contains the new 
password. When the Agent 1 08 gets around to process- 
ing this request, it re-encrypts the user's account entry 
in the User Account Database file 146 according to the 
new password and acknowledges the request by updal- 3c 
ing his Password Log file (e.g., 140). m the interim, the 
user may to use the new password, because a list of 
recent passwords is maintained in the Password Log 
file, encrypted using the latest password. When the user 
needs access to the userDirKey 541 (e.g., to perform a 35 
restore), the software uses the latest password to 
access userDirKey 541 in the User Account Database 
146; upon failure, the password "history" is accessed 
and oW passwords are tried automatically until one 
works. Thus, the user can change his password several 40 
times and continue to work without needing to wait for 
the Agertt 108 to process his change request Noted 
that CRCs are embedded in these files in all cases to 
verify that the password is correa In the worst case of 
a user forgetting his passwad or inadvertently deleting -« 
his Password Log file while a request is pending, the 
administrator can easily issue the user a new password. 

As an administrator-configuraWe option in the pre- 
ferred embodiment, in order to help insure a certain 
level of security, the backup software may pronpt the so 
user to change his password on a periodic basis and 
check that all passwords have a minimum length (and 
are not re-used). In an alternate embodiment, as an ulti- 
mate backdoor, it would be possible to have the admin- 
istrator software keep a log of all user passwords and 55 
userDirKey values in a file that is encrypted using a pub- 
lic key algorithm, which only a certified third party has 
the ability to decrypt. In this case, if the administrator 
loses the ability to restore passwords, the third party 



could recover the administrator and user passwords, 
probably for a considerable fee to cover the cost of 
checking the legitimacy of the request and to discour- 
age frivolous use of this service. 

One goal of the preferred embodiment is to allow 
the user to perform backups without entering a pass- 
word. This ability is particularly, important in the com- 
mon case of performing scheduled backups when the 
user is not present. At the same time, it is dearly desir- 
able to require a password in order to restore data. For- 
tunately, this feature is easily implemented as follows. 
During each backup, the backup software posts the 
backup directory file (e.g.. 143), encrypted using a spe- 
cial user-specific key (userPostKey) just for this pur- 
pose. The userPostKey value is included in the user 
account entry (which is encrypted using the user pass- 
word 540) of the User Account Database file 146; this 
key may also be stored on the k3cal workstation disk so 
that it is available without entering a password. As part 
of the migration of the backup directory file to the 
\BACKUP\SYSTEM path 122. the Agent 108. which has 
access to both keys, subsequently re-encrypts this file 
using userDirKey 541. In the preferred entxxJiment, 
there is thus a brief period of time, from when the 
backup directory file is first posted unti the Agent 108 
migrates it, when the system is dependent on network 
security and on the security of the local workstation to 
maintain the privacy of the backup sat, since a hacker 
could in theory copy the userPostKey from the local 
workstation and the backup directory file (e.g., 143). It 
wouW be possible to overcome this limitation in an alter- 
nate embodiment by posting the directory file with a 
pii)lic-key encryption algorithm, using the Agent's pub- 
lic key; such an approach seems overkill, however, par- 
ticularly in light of the fact that once a hacker has access 
to the user's workstation (to get the unauthorized copy 
of userPostKey). the privacy, of the backi^j data set is 
probably the least of anyone's concerns. 

In addition, the backup software maintains the Pre- 
vious Dir file (e.g., 141). which is also encrypted with 
userPostKey and can thus be accessed without a pass- 
word. This file comains a copy of all the directory infor- 
mation for the most recent badaip. allowing 
identification of unchanged and modified files at the 
next backup. The software of the preferred embodiment 
may also retain a cached copy of this file on the local 
workstation to minimize network bandwidth. Note that, 
since this file does not contain the encryption finger- 
prints that are used for encrypting the file data, only a 
knowledge of directory information (as opposed to the 
file data encryption keys) would be compromised in the 
worst case H the contents of the Previous Dir file were 
somehow compromised. In the rare case where this file 
is corrupted or deleted, which can be detected by 
checking CRCs. the backup software of the preferred 
embodiment rebuilds the Previous Dir file from the pre- 
vious (encrypted) backup directory file(s). although 
such rebuilding does require the user to enter his pass- 
word. 



20 



39 epot: 

3. Restore Process 

The preferred embodiment provides two principal 
ways oi selecting the tackup set to be restored. In the 
conventional method, the user is presented with the list 
of previous backup operations, each identified with the 
backup time, date, and description (e.g., from a user's 
Backup Log file such as 150), from which he selects the 
desired backup set. In the alternate approach, the user 
selects a file from the current disk contents and is pre- 
sented with a list of all previous versions of that file con- 
tained in all the backup sets. This list is typically 
presented as a seieaaWe set of icons on a calendar 
showing when new versions were backed up In order to 
speed up the initial generation of this list once the user i 
has chosen the file, in an alternate embodiment, a 
Jasiversron- field is added to each (tteE„j^, record 207 to 
provide a direct linked list of all unique versions of each 
file, as mentioned previously 

In the preferred embodiment, there are two melh- 2 
ods of restoring data from the backup storage moans 
101 once the backup set is selected. The first technique 
is basically klentical to a "conventional" restore opera- 
■ tion. The user is presented with a tree of files available 
for restore, where the directory information is extracted a 
from the associated backup directory file. After the user 
"tags" the desired files and specifies the restore desti- 
nation, the restore software retrieves the file contents 
from the backup data file(s) and writes them to the des- 
tination. 3j 

The second restore paradigm provides much more 
flexibility in accessing the data. Once the user selects 
the backup set, the file set is "mounted" as a read-only 
disk volume by a special file system driver. This driver is 
implemented as an installable file system (IFS) in the J5 
preferred embodiment; in an alternate embodiment, the 
disk volume is mounted using a block device driver in 
which the on-disk format of a normal disk volume is syn- 
thesized to match the contents of the tjackup set. 
Regardless of its underlying structure, tfie driver pro- 40 
vides all the operating system specific functons neces- 
sary to allow any application to access the files. For 
example, if the user wishes to view a spreadsheet file 
that was backed up in the associated backup set, once 
the backup set is mounted he may simply run his 45 
spreadsheet program and open the file directly on the 
mounted volume, without having first to copy the file to a 
local hard disk; afternately, the user may sinply copy 
any files from the mounted volume to his local hard disk 
using his own favorite file management application. This sc 
approach allows the user to access his backi43 data in a 
more intuHive way, using his own tools and applications 
instead via a dedicated restore application that is unfa- 
miliar because it is rarely used. It also works around the 
common problem of inadvertently ovemvriting the cur- ss 
rent version of a file when restoring an older version 
from a backup set using a conventional restore pro- 
gram. 

Observe that, because the backup storage means 
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101 is a random access device, the time required to 
access any file is comparable to typical disk access 
times, although it may require a few more seeks to tol- 
'o* lexternPtr' references 420. The associated backup 
5 directory file is loaded from disk very quickly once the 
backup set is chosen, after which the access to any par- 
ticular file anywhere in the backup direaory tree 
involves only reading in the associated ^„ieinio' record 
408 and accessing the data blocks. Thus, a restore 
10 operation in the preferred embodiment is considerably 
faster in almost all cases than a comparable restore 
operation from a tape backup system. In particular, file 
access is fast enough that accessing files on the 
mounted backup volume is usually imperceptibly slower 
5 than accessing the files on the original disk drive! An 
alternate embodiment can take further advantage of 
this "real-time" nature of the mounted backup volume by 
adding driver software logic allowing it to be writable, in 
which all writes actually are stored in a local fransient 
5 cache that may overflow onto the local disk. Any writes 
to this transient cache will be discarded once the vol- 
ume is unmounted, Such an approach allows the user, 
for exanple, to mount a volume and perform a transient 
"update in place" operation, such as a compilatton or a 
: database sort, retrieve the relevant results from the 
operation, and then unmount the volume: effectively, the 
user has temporarily taken his disk drive back in time to 
perform the update operation. 

The restore method of the preferred entKxiiment is 
also somewhat unique in that, although each backup 
operation after the initial backup is effectively an -incre- 
mental" backup, the image presented for restore con- 
tains all files present on the source disk at the time of 
the backup, and all of these files are accessible in real- 
time, as discussed above. The random access nature of 
the backup staage means 101 altows only file changes 
to be stored, thus providing great savings in storage 
cost, while still allowing for real-time access to all files. 

Another msgor benefit accrues to the present inven- 
tion. Note that, once the system is installed and config- 
ured, no administrator interaction is required to perform 
any backup or restore operation, other than to make 
sure that there is sufficient free disk space on the 
backup storage means 101. Assuming that the cost of 
providing enough disk space is acceptably low, which 
seems to be the case in practice due to the high levels 
of data reduction achieved in the preferred errtiodiment. 
the backup system has a very low maintenance cost. By 
comparison, most tape backup systems require opera- 
tor intervention to change tapes periodically and to 
mount a tape for a given restore request: even with 
(expensive) tape or optical disk jukebox hardware, such 
operations seem almost primitive in contrast with the 
real-time nature of the present invention. 

Note that, in an alternate embodiment, the present 
invention can also be applied to a single computer. In 
this case, the backup staage means 101 might be a 
section of the local hard disK or a removable disk 
device (eg., Bernoulli, Syquest), or a portion of a net- 
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work disk volume. The advantages of duplicate file iden- 
tification probably are not significant in this instance, but 
all the other considerable benefits discussed, still apply. 
The Agent process 108 could be ruin as a background 
process on the single computer, with the user acting as 5 
tne backup administrator, or the Agent functions could 
be configured to run automatically as part of each 
backup operation. 

4. Agent Functions re 

The Agent process 108 runs on a node 107 on the 
network 106, typically as a background task on a desk- 
lop PC, but it may also run as a software task on the fie 
server 100. The backup administrator configures the is 
Agent process 108, both in its location and performance 
characteristics, which are quite scalable, as described, 
below. These settings may be varied over time as use of 
the backup system e/olves. For example. In a backup 
system with only a few users, the Agent process 1 08 sc 
may run as a background task on the administrator's 
own desktop PC. As more users are added and the 
Agent process 108 requires more time, the administra- 
tor may opt to dedicate a PC on the network to run the 
Agent process 108. Eventually, it may make sense to 25 
install a backup file server dedicated solely to backup, 
including running the Agent process 108. which then 
can access the backup storage means 101 as a local 
disk volume instead of over the network. It is fairly sim- 
ple in the prefen-ed embodiment to change how often so 
and where the Agent process 108 runs in order to meet 
the needs of the backip dients. 

In addition to the migration and other Agent func- 
tions discussed above, there are some other concerns 
that must be addressed by the Agent software. For 35 
example, there is a potential problem with backup "cli- 
ents" that crash during the middle of a backup opera- 
tion. Similarly, the Agent 108 itself could crash during a 
migration or consolidation (discussed below) operation. 
Both the application and Agent software are robust « 
enough in the preten-ed embodiment to detect such 
conditions and respond properly, including the ability to 
'clean up their own mess" the next time they are run. A 
few other such issues are discussed briefly below. 

With a little thought, it becomes clear that there is a 45 
small problem in the preferred embodiment which, if 
ignored, might cause a badaip operaton to fail to iden- 
tify some duplicate files and thus slightly affect storage 
requirements. If two users are performing backups con- 
cun-ently (or actually if one starts before the other's so 
backup files have been migrated by the Agent 108 from 
the \BACKUP\USER path 121 to the \BACKUP\SYS- 
TEW path 1 22), neither user will be able to identify dupli- 
cate files from the other. This is probably of most 
concern during the "initialization" period that occurs 55 
when the first few users are running their initial backup, 
though the problem never goes away emirely. The work- 
around for this problem in the preferred embodiment is 
to have the Agent 108 perform some additional dupli- 
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cate file "elimination' as part of the migration process. 
This can be done without modifying the contents of the 
backup directory files: instead, the <fiioin(o' entry 408 tor 
a duplicate file is changed to contain a single (8«arn=tr- 
reference 420 encompassing the entire file. For per- 
formance reasons, this activity might actually be 
deferred until a later time, such as the middle of the 
night, when the network should have less traffic. It is 
possible in practice that this problem simply isn't signifi- 
cant enough to worry about, particularly if the adminis- 
trator "primes the pump' after installation by having a 
few representative nodes perform their initial backups 
sequentially to buikJ up the initial gtobal database. Thus, 
in the preferred embodiment, the administrator can dis- 
able this functionality. 

In some cases, a user may wish to delete certain 
backup sets, typically to save space on the backup stor- 
age means 101. For exanple, the user may decide to 
merge old daily backups into weekly (or monthly) back- 
ups after a few months have passed. Because of the 
duplicate file identrtication and file differencing of the 
preferred embodiment, the resulting disk savings are 
usually fairiy small. In the preferred embodiment, the 
backup application posts a file requesting the Agent 1 08 
to perform the deletion, which may involve consolidating 
several backup sets into a single backip directory/data 
file set in order to retain copies of any file and directory 
entries that are referenced by the remaining backup 
sets, either of this user or other users. This consolida- 
tion operation may be best deferred until a non-busy 
time on the network. Completion of the consolidation 
operation may also have to be deferred until no users 
have a backup set mounted that contains a reference to 
the f ile{s) in questions. Observe that the use of indices 
(instead of direct pointers) both for file and directory ref- 
erences greatly simplifies such an operation; such con- 
solidation could still be performed without this extra level 
of indirection, but it would in general involve time-con- 
suming changes to many of the remaining backup files, 
instead of the creation of the single "stub" backup file 
set that results in the preferred embodimerrt. 

5. Administrator functions 

The liackup administrator, who may or may not be a 
network administrator, has several functions to perform 
in the preferred emtwdiment. It is intended that these 
functions be largely transparent, with most effort being 
expended at installation time, but from time to time other 
decisions and actions may become necessary. 

5.1 Installation 

The baci<up administrator should install the backup 
software on the Agent computer 107 (which may be his 
own desktop) and should set ip the network direaory 
structure (e.g., VBACKUP\SYSTEM 122 and 
\BACKUP\USER 121) where the backup files are to be 
stored. Setting up the inital directory structure arxl 
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access rights may involve some help from a network 
administrator, depending on the network access rights 
of the backup administrator. 

The backup softvvare is distributed on either CD- 
ROM or floppy disk, but in general only the backup 
administrator will ever have to use the distribution 
media, since the software of the preferred embodiment 
installs itself on the network in such a way to allow users 
to run a SETUP program from ttie network. To as large 
an extent as possible, the installation is automated: the 
administrator only has to intorm ttie installation soflvirare 
where the \BACKUP directory 120 is located, and the 
software Installs itself. 

5.2 Adding new users 

With the backup software installed, before a user 
can actually perform any backups, ttie administrator 
must set up an "account" for the user in the User 
Accourrt Database 146. This is important for two rea- 
sons. First, each user must have his own directories 
(e.g., 125, 129) and a unique (ussrindex) number which is 
crucial In identifying files that are shared across users. 
Second, keeping an account database allows the 
administrator to limit access to the system and to meter 
use of the software according to the terms of his license. 

As part of adding a new user account in the pre- 
ferred emtKxfiment the administrator software creates 
the user directories with appropriate access rights 
(again, this may require network administrator rights). 
Each user is also assigned a unique user name ctrasen 
by the Administrator, such as JOHN, and a unique 16- 
^'^ (userindexi (which the user never needs to know 
directly). This information, together with the unique user 
directory name (e.g., USER2), which is based on the 
(userindsx) instead of the user name in the pretended 
embodiment, is written to the User Account Database 
146, which is read-only for all users. Note that, although 
the presence of this User Account Database does allow 
a hacker to determine the (ussrindsx) and directory name 
of any user (with some reverse engineering, since those 
two fields, while not encrypted, are not stored in the 
clear in the preferred embodiment), such kncjwiedge 
does not compronise the privacy of the user's data in 
any way other than perhaps a knowledge of the fre- 
quency and size of backup sessions, assuming the 
user's password is not compromised The administrator 
also assigns the user an initial password wrhich is used 
to encrypt pnvate fields of the user's account, such as 
the userDirKey and userPostKey values. In the pre- 
ferred embodiment, the user account entry is duplicated 
in the User Account Database 146, encrypted with the 
administrator's passvrord, so that the keys will not be 
lost if the user forgets his password. 

The administrator next informs the user, usually via 
e-mail, that his account is now active, giving him the 
assigned user name and (temporary) password. The 
user then runs the SETUP program from the 
\BACKUP\SYSTEM\GLOBAL directory, which under 



Microsoft Windows 3 1 may be effected by attacning a 
.EXE file to the e-mail message so that the user can 
simply double-click the icon The user enters his 
account name and password, and the software sets up 

5 a personal backup directory (typically on the user s local 
hard disk) and copies over any necessary files to tnat 
directory. This personal directory is also used tor cach- 
ing certain tiles, such as the Previous Dir file (e.g. , 141 ), 
in order to minimize network bandwidth consumption, 

10 Note that it is possible (and probahdy desirable), if the 
user so chooses, to copy only a minimum set ot pro- 
gram files locally, so that the user always runs the latest 
copy of the software from the network. Alternately, the 
software checks its version against that on the network 

IS to make surer that it is the latest and ask the user for 
permission to upgrade when a new version is detected. 

The user may also be asked to change his personal 
password during the initial installation. During the 
SETUP procedure, the user will be queried to enter any 

20 relevant personal preferences, such as how often to 
schedule periodic backups and where the personal 
tiackup directory shoukJ be located. These preferences, 
along with the user name and (usorindeni' are stored in 
the User Preferences file (e.g.. 142). Most preferences 

25 may be changed later. 

5.3 Overseeing the Agent 

The Agent task in general runs in the background 

30 without any supen/ision. However, circumstances may 
arise (such as a system crash) that could require some 
intervention by the administrator to restart the Agent 
process 108. It is intended that the Agent 108 in general 
be at)le to recover from most problems that arise, but it 

35 is probably not possible to guarantee complete recover- 
atiility The Agent process 108 of the preferred embodi- 
ment generates a log file of Hs activities that the 
administrator can review. The administrator also has a 
monitoring application that can perform some simple 

•w checks to make sure that the Agent 1 08 is performing its 
tasks on a timely basis and in a reasonable fashion, giv- 
ing warnings upon observing any activity (or lack 
thereof) that appears suspect. 

The invention has been desaibed in an exemplary 

45 and prefen-ed embodiment, but is not limited thereto. 
Those skilled in the art will recognize that a number of 
additional modifications and improvements can be 
made to the invention without departure from the essen- 
bal spirit and scope. The scope of the invention should 

so only be limited by the appended set of daims. 

Claims 

1 . A method for backing up data files stored on a disk 
£5 volume of a node of a computer network to a 
backup storage means, said backup storage means 
containing data files already backed up from other 
nodes on said computer netwak, said method 
connprising the steps of; 
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searching through a list of said files already 
contained In said backup storage means for a 
match to files to be backed up from said disk 
volume; 

operative when no match is found between a s 
file to be backed up from said disk volume and 
any of said files already contained in said list, 
storing on said backup storage means a com- 
plete representation of the contents of said file 
to be backed up, computing an index that indi- ic 
cates the location on said backup storage 
means of said complete representation, and 
adding to said list an entry desaibing said file 
to be backed up from said disk volume: 
operative virhen a match is found between a fie rs 
to be backed up from said disk volume and a 
file already contained in said list, computing an 
index that indicates the location on said backup 
storage means of a complete representation of 
the contents of said file already contained in so 
said list; 

storing a data structure specifying the directory 
structure of said disk volume at the time of the 
backup operation, said data structure also 
including, for each said file backed up from said 25 
disk volume, said index indicating the location 
of said complete representation, either of said 
file to be backed up or of said file already con- 
tained in said list, depending on the outcome of 
said search through said list; and ac 
whereby a file ttiat is duplicated across nodes 
may be identified so that only one copy of the 
contents of said file is stored on said backup 
storage means. 

35 

The method of claim 1 in which the step of storing 
said complete representation of the contents of said 
file to be backed up further includes the step of: 

operative when a previous version of said fie 40 
has already been backed up from said node to 
sakJ backup storage means, computing the dif- 
ferences from the previous version of said file, 
representing portions of the contents of said 
file to be backed up using indices into the rep- 4s 
resentation of said previous version on said 
backup storage means. 

The method of claim 2 in which the existence of 
said previous version of said file is detected using a so 
previously saved data structure specifying the 
directory structure of a previous backup operation, 
and in which said differences between said ver- 
sions are computed using an index, contained in 
said previously saved data structure, to a complete 55 
representation of the contents of said previous ver- 
sion of said file. 

The method of claim 3 in which the steps of storing 



said complete representation of the contents of said 
file to be backed up further includes the step of 
compressing portions of said representation using 
a lossless data compression algorithm before stor- 
ing said representation on said backup storage 
means. 

5. TTie method of claim 4 in which the step of storing 
said complete representation of the contents of said 
file to be backed up further includes the step of 
encrypting portions of said complete representation 
using a hash encryption key that is derived by com- 
puting a hash function on the contents of said file to 
be backed up. 

6. The method of claim 5 in which saki hash encryp- 
tion key is stored as part of said data structure 
specifying the directory structure of said disk vol- 
ume. 

7. Tfia method of claim 1 in which the step of storing 
said complete representation of the contents of sakJ 
file to be backed up further includes the step of 
encrypting portions of said complete representation 
using a hash encryption key that is derived from a 
confuting a hash function on the contents of said 
file to be backed up. 

8. The method of daim 7 in which said hash enayp- 
tion key is stored as part of said data structure 
specifying the directory structure of said disk vol- 

9. The method of any of claims 1-8 in which the step 
of storing said data structure specifying said struc- 
ture of said disk volume further includes the steps 
of 

conpressing portions of said data structure 
with a lossless data compression algorithm; 
and 

encrypting sakI hash encryption key using an 
encryption key that is private to sakl node on 
said computer network. 

10. The method of any of daims 1-8 in which the said 
list of said files already contained in sakl backup 
storage means is organized as a database in order 
to minimize search time. 

1 1. The method of claim 10 in which each entry in said 
database includes a bash function computed on the 
directory entry information for the file associated 
with said entry including the file name, length, and 
time of creation, and a hash function computed 
over portions of the contents of said file. 

1 2. The method of claim 1 1 in which said search of said 
database includes the following steps: 
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loading a first section of said database, said 
first section containing partial entries, each 
partial entry containing only a portion of an 
entry of said database; 

generating a new database entry of said file to 5 
be backed up; "and 

searching through said first section for a match 
between said new database entry and said par- 
tial entries and, operative when a match is 
found between said new database entry and a i« 
partial entry of said first section, loading the 
remaining portions of said matching partal 
entry from the associated entry in a second 
section of said database, and comparing said 
new database entry with said remaining portion is 
of said associated enUy to determine whether 
there is a complete match between said new 
database entry and the complete database 
entry 

20 

1 3. The method of claim 12 in which said first section of 
said database is stored in a sorted order based on 
bit fields of said partial entries and is compressed 
with lossless data compression algorithm. 

25 

14. The method of claim 13 in which said lossless data 
compression algorithm includes storing an array 
indicating how many consecutive entries in said 
sorted first section have bit fields of each possible 
value of said bit fields, and in which the rest of said 3o 
first section omits said bit fields from the remainder 

of said partial entries. 

15. The method of any of claims 1-8 in which the con- 
tents of a particular backup operation are mounted 3S 
as a restored disk volume having a directory struc- 
ture identical to that of the original disk volume at 

the time of said backup operation, whereby said 
files on said restored disk volume may be accessed 
from any application software that uses normal file 40 
system input/output calls. 

16. The method of claim 15 in which sakf restored disk 
volume is accessible on a read-only basis. 

45 

17. The method of claim 15 in which said restored disk 
volume is accessible for reads and write, all writes 
to said restored disk volume being cached in a tran- 
sient storage means, the contents of which are dis- 
carded when said restored disk volume is so 
unmounted. 

18. The method of any of claims 2-8 in which the differ- 
ences between said file to be backed up and said 
previous version of said file are computed using a 55 
probabilistic algorithm, including the following 
steps: 

at the time when said previous version was 
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backed up, storing on said backup storage 
means a set of hash function values computed 
on fixed size chunks of said previous versions: 
at the time of said backup, loading said previ- 
ously stored hash function results: 
comparing said hash function results from said 
previous file version to hash function results 
computed on fixed size chunks of said file to be 
backed up; and 

operative when a chunk of said file to be 
backed up has the same hash value as a chunk 
of said previous file, representing sad chunk of 
said file to be backed up by an index indicating 
sakJ matching chunk of said previous version. 

19. The method of claim 18 in which said comparison 
of said hash function results includes sliding the 
hash function computation from byte to byte within 
said file to be backed up, whereby matching chunks 
in said file to be backed up may be found on any 
byte txjundary in said file to be backed up. and not 
solely on chunk boundaries. 

20. The method of claim 1 9 in which said hash function 
includes computation of a cyclic redundancy check 
(CRC). 

21. The method of daim 18 in which the contents of a 
particular tackup operation are mounted as a 
restored disk volume having a directory structure 
identical to that of the original disk volume at the 
time of said backup operation, whereby said files on 
said restored disk volume may be accessed from 
any application software that uses normal file sys- 
tem input/output calls. 

22. The method of daim 21 in v*ich said restored disk 
volume is accessible on a read-only basis. 

23. The method of daim 22 in which said restored disk 
volume is accessible for reads and write, all writes 
to said restored disk volume being cached in a tran- 
sient storage means, the contents of which are dis- 
carded when said restored disk volume is 
unmounted. 
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